Use bgipfs for upload #1039

azf20 · 2025-01-25T23:35:05Z

Description

To upload a site to BuidlGuidl's IPFS node, in the nextjs package:

yarn ipfs:init
yarn ipfs

These commands wrap initialisation and upload commands from bgipfs. This uses a default proxy endpoint (no auth required at this time)

technophile-04

This is great Adam!! was just testing and while running yarn ipfs I got the below error:

ipfs error

󰂃 6% ❯ yarn ipfs
  ▲ Next.js 14.2.11

   Creating an optimized production build ...

🌼   daisyUI 4.12.10
├─ ✔︎ 2 themes added             https://daisyui.com/docs/themes
╰─ ❤︎ Support daisyUI project:   https://opencollective.com/daisyui

 ✓ Compiled successfully
 ✓ Linting and checking validity of types
 ✓ Collecting page data
   Generating static pages (0/8)  [=   ](node:6527) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:6534) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:6536) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:6535) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:6537) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:6533) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
 ✓ Generating static pages (8/8)
 ✓ Collecting build traces
   Finalizing page optimization  ..(node:6609) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
 ✓ Finalizing page optimization

Route (app)                                                                  Size     First Load JS
┌ ○ /                                                                        1.65 kB         432 kB
├ ○ /_not-found                                                              879 B          89.7 kB
├ ○ /blockexplorer                                                           1.4 kB          435 kB
├ ● /blockexplorer/address/[address]                                         235 B           434 kB
├   └ /blockexplorer/address/0x0000000000000000000000000000000000000000
├ ● /blockexplorer/transaction/[txHash]                                      1.42 kB         432 kB
├   └ /blockexplorer/transaction/0x0000000000000000000000000000000000000000
└ ○ /debug                                                                   7.35 kB         438 kB
+ First Load JS shared by all                                                88.8 kB
  ├ chunks/7023-38a60bcaec288bbe.js                                          31.9 kB
  ├ chunks/fd9d1056-805ccda7fca25131.js                                      53.6 kB
  └ other shared chunks (total)                                              3.32 kB


○  (Static)  prerendered as static content
●  (SSG)     prerendered as static HTML (uses getStaticProps)

IPFS add error: {
  error: HTTPError: Request Entity Too Large

  FUNCTION_PAYLOAD_TOO_LARGE

  bom1::vhn88-1737877490716-8ee1864a251b

      at Object.errorHandler [as handleError] (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/kubo-rpc-client/dist/src/lib/core.js:81:17)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async Client.fetch (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/kubo-rpc-client/dist/src/lib/http.js:80:21)
      at async KuboRPCClient.addAll (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/kubo-rpc-client/dist/src/add-all.js:21:25)
      at async Object.directory (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/ipfs-uploader/dist/NodeUploader.js:90:42)
      at async UploadCommand.run (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/bgipfs/dist/commands/upload/index.js:50:19)
      at async UploadCommand._run (/Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/@oclif/core/lib/command.js:312:22)
      at async Config.runCommand (/Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/@oclif/core/lib/config/config.js:435:25)
      at async run (/Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/@oclif/core/lib/main.js:96:16)
      at async file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/bgipfs/bin/run.js:5:1 {
    response: Response {
      status: 413,
      statusText: 'Request Entity Too Large',
      headers: Headers {
        'cache-control': 'public, max-age=0, must-revalidate',
        'content-length': '93',
        'content-type': 'text/plain; charset=utf-8',
        date: 'Sun, 26 Jan 2025 07:44:52 GMT',
        server: 'Vercel',
        'strict-transport-security': 'max-age=63072000',
        'x-matched-path': '/api/ipfs-proxy/add',
        'x-vercel-error': 'FUNCTION_PAYLOAD_TOO_LARGE',
        'x-vercel-id': 'bom1::vhn88-1737877490716-8ee1864a251b'
      },
      body: ReadableStream { locked: true, state: 'closed', supportsBYOB: true },
      bodyUsed: true,
      ok: false,
      redirected: false,
      type: 'basic',
      url: 'https://www.bgipfs.com/api/ipfs-proxy/add?stream-channels=true&wrap-with-directory=true&cid-version=1&progress=false'
    }
  },
  message: 'Request Entity Too Large\n' +
    '\n' +
    'FUNCTION_PAYLOAD_TOO_LARGE\n' +
    '\n' +
    'bom1::vhn88-1737877490716-8ee1864a251b\n',
  stack: 'HTTPError: Request Entity Too Large\n' +
    '\n' +
    'FUNCTION_PAYLOAD_TOO_LARGE\n' +
    '\n' +
    'bom1::vhn88-1737877490716-8ee1864a251b\n' +
    '\n' +
    '    at Object.errorHandler [as handleError] (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/kubo-rpc-client/dist/src/lib/core.js:81:17)\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n' +
    '    at async Client.fetch (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/kubo-rpc-client/dist/src/lib/http.js:80:21)\n' +
    '    at async KuboRPCClient.addAll (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/kubo-rpc-client/dist/src/add-all.js:21:25)\n' +
    '    at async Object.directory (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/ipfs-uploader/dist/NodeUploader.js:90:42)\n' +
    '    at async UploadCommand.run (file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/bgipfs/dist/commands/upload/index.js:50:19)\n' +
    '    at async UploadCommand._run (/Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/@oclif/core/lib/command.js:312:22)\n' +
    '    at async Config.runCommand (/Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/@oclif/core/lib/config/config.js:435:25)\n' +
    '    at async run (/Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/@oclif/core/lib/main.js:96:16)\n' +
    '    at async file:///Users/shivbhonde/Desktop/github/scaffold-eth-2/packages/nextjs/node_modules/bgipfs/bin/run.js:5:1'
}
{
  success: false,
  cid: '',
  error: 'Request Entity Too Large\n' +
    '\n' +
    'FUNCTION_PAYLOAD_TOO_LARGE\n' +
    '\n' +
    'bom1::vhn88-1737877490716-8ee1864a251b\n'
}
 ›   Error: Upload failed: Upload failed

azf20 · 2025-01-26T12:48:01Z

thanks @technophile-04! that's hitting a vercel function size limit, pushed a new approach which is using a cf worker

portdeveloper · 2025-01-27T07:47:45Z

Everything works great! Amazing work @azf20 !!!

This is a lot faster than my code and it is cleaner as well!

Here is the link to my app deployed with Adam's code: https://bafybeiduwahy6nmpkpvcebtctyplxmdgjvlrpwkcnrzi6zqaizse64muna.ipfs.gateway.bgipfs.com/

And here it is after adding it to my ens records: https://pallas-athena.eth.limo/

And it just works!

technophile-04 · 2025-01-27T11:48:08Z

THanks for the updates!! Yes works great now!!

Can you run yarn install once and push those commits? Because when I switch to this branch and do yarn isntall it gives me this changes:

changes image

Also a question, is there a way we could make this just on command? Instead of it developer needing to run two separate commands? Like just yarn ipfs should do it all?

Maybe we just un-gitignore the ipfs-upload.config.json? and ship that file with SE-2 directly? Since I see there is no app specific config init and seems same config for all app? (I am sure I am missing something)

Small improvement nitpicks:

While logging the link from bigipfs CLI can we log the whole link in the end?

So that the developer doesn't need to construct the whole link himself manually?

Another nitpick maybe we could handle it in future not at all necessary now:

Unfurl are not resolving:

App link: https://bafybeihijk22a4rnuxrv26ulacy3r3qzkjcbnt25atg7aqnlwqmyddblk4.ipfs.gateway.bgipfs.com/

azf20 · 2025-01-28T14:41:16Z

OK great! Been making some changes to the underlying infrastructure so let's coordinate when this rolls out!

Pabl0cks

Great job @azf20! This is really awesome 🔥🙏

I hate to be the "Windows guy" here... tried to run yarn install and is throwing some errors :

Error on Git Bash when trying to `yarn install`

➤ YN0001: │ Error: ENOENT: no such file or directory, open
    at Object.open (node:fs:565:11)
    at Object.writeFile (node:fs:2309:6)
    at node:internal/util:442:7
    at new Promise (<anonymous>)
    at Object.writeFile (node:internal/util:428:12)
    at xze (E:\GitHub\scaffold-eth-2\.yarn\releases\yarn-3.2.3.cjs:236:193779)
    at async Promise.all (index 0)
    at async UL (E:\GitHub\scaffold-eth-2\.yarn\releases\yarn-3.2.3.cjs:236:1927
49)
    at async Gze (E:\GitHub\scaffold-eth-2\.yarn\releases\yarn-3.2.3.cjs:710:128
75)
    at async Oze (E:\GitHub\scaffold-eth-2\.yarn\releases\yarn-3.2.3.cjs:710:118
46)
➤ YN0000: └ Completed in 30s 834ms
➤ YN0000: Failed with errors in 38s 447ms

If I try to install bgipfs package manually, I get these error logs:

2025-02-02T18_17_04_844Z-debug-0.log

When I use WSL on my Windows machine, yarn install + yarn ipfs are working out of the box 👌:

✓ Configuration file initialized successfully.
✓ File uploaded. CID: bafybeidxejqgov2pso47erm4uzjxa2ddo2jao7ge2ews22mvwdi5m2jepi
🚀 Upload complete! Your site is now available at: https://gateway.bgipfs.com/ipfs/${cid}

Just as nitpicks:

cid is not getting replaced (maybe is a WSL issue)
The gateway url doesn't seem to work when I try to open it on chrome/brave

azf20 · 2025-02-03T13:49:51Z

hey thanks so much @Pabl0cks! Good catch, the gateway url hadn't been updated, and it was just printing {cid} which I agree is less friendly than it might be - pushed a fix for both of those things. I'm not sure what is required for better Windows support

portdeveloper · 2025-02-04T05:48:15Z

I believe that we should focus only on WSL for windows support as everyone is moving away from git bash to WSL.

carletex · 2025-02-04T10:09:16Z

I believe that we should focus only on WSL for windows support as everyone is moving away from git bash to WSL.

I'm not against this! Seems like a pretty common practice and will save us from a lot of trouble. We'd just to state it clearly in the Docs.

Let's see what others think too

packages/nextjs/scripts/deploy-ipfs.mjs

technophile-04

Works like a charm! Thanks @azf20! Just requested a small changes at https://github.com/scaffold-eth/scaffold-eth-2/pull/1039/files#r1944592425 but other than that everything looks great!

I'm not against this! Seems like a pretty common practice and will save us from a lot of trouble. We'd just to state it clearly in the Docs.

Yup I think mentioning in docs should be sufficient 🙌

rin-st

Works great for me too!

I'm not against this! Seems like a pretty common practice and will save us from a lot of trouble. We'd just to state it clearly in the Docs.

Agree!

technophile-04

Ok this is ready to go! Tested it on fresh clean install with the branch works great too, also tested with .env.local and it picks correct env variables too 🙌 So yeah we just need to push the red button to launch this 🙌

I was trying to compare the install times of this branch and main branch and its 2-3 sec difference with yarn.lock present :

test 1:

test 2:

carletex · 2025-02-11T10:13:04Z

Let's wait until @Pabl0cks test it again with Windows WSL....and maybe add the WSL requirement to the README?

Pabl0cks · 2025-02-11T14:09:03Z

For WSL, it works great. I tried it again on my desktop, which didn't have WSL, and it worked nicely.

This has been lingering in the back of my mind for a while. I'm afraid of "blocking" some Windows users from SE-2, as their yarn install would fail. I'm still hesitant to force them to fully commit to WSL, even though I agree it’s much handier for using some of the popular tools and technologies (e.g., Foundry, Eliza).

I've been testing and found that if I install the buidlguidl-ipfs repo locally and update the package.json to point to my local path, it works:
"bgipfs": "file:E:/GitHub/buidlguidl-ipfs", then yarn install works fine (even tho it takes 23m to complete, no idea why).

GPT is suggesting me that probably the files section of bgipfs package.json should not have the initial "/" to be working on Windows.

Modify the Files Field:
 – In the bgipfs package.json, remove the leading slashes from the directory entries. Instead of
   
text
"files": ["/bin", "/dist", "/templates", "/oclif.manifest.json"]

  use
   
text
"files": ["bin", "dist", "templates", "oclif.manifest.json"]

 This change ensures that the files and directories are correctly picked up during publication on all platforms.

Could this make sense @azf20? What would be the easier way to try it out? A bit npm-newbie here, but happy to help 🙏

Even if the origin is not exactly that one, it may be some other parameter of the package.json it seems.

azf20 · 2025-02-19T11:20:13Z

thanks so much for taking a look @Pabl0cks! I made changes and pushed with the latest version of bgipfs, hope that works on windows now!

Pabl0cks

It was still failing with the latest changes, but today @technophile-04 helped me debug the problem and we (he 😅) found out the reason!

bgipfs requires node version >20.18.1 and i was using 20.11

Updated my node version and is working nicely, both yarn install & yarn ipfs. Awesome job everyone! Approving & merging this

carletex · 2025-02-21T11:03:31Z

Great!

bgipfs requires node version >20.18.1 and i was using 20.11

Then we need to update the project dependencies on the README / docs, right?

portdeveloper and others added 11 commits January 9, 2025 10:45

Add config for yarn ipfs

3da1015

Show errors in yarn ipfs

5fef988

Fix formatting in next.config.js

470f9c1

Update deploy-ipfs to use nifty ink

1547922

Change deploy-ipfs.js to deploy-ipfs.mjs

6026051

Remove unused stuff from yarn.lock

f7cbc7c

Fix build flag not getting detected

0c139a0

Merge branch 'main' into feat/ipfs-manual

8f06d03

Merge branch 'main' into feat/ipfs-manual

faa4150

feat: bgipfs for upload

247edb4

using proxy instead of auth

b8a3a53

technophile-04 reviewed Jan 26, 2025

View reviewed changes

replace with community endpoint (uses cf worker)

9f56153

azf20 added 2 commits January 31, 2025 16:22

update + one command

3efe720

use latest endpoint

0a8de9e

Pabl0cks reviewed Feb 2, 2025

View reviewed changes

fix url & pass through cid

b215ff9

technophile-04 changed the base branch from feat/ipfs-manual to main February 6, 2025 11:43

technophile-04 added 2 commits February 6, 2025 17:17

yarn install changes

ea4d9a6

resolve merge conflict

aafd627

technophile-04 reviewed Feb 6, 2025

View reviewed changes

packages/nextjs/scripts/deploy-ipfs.mjs Outdated Show resolved Hide resolved

technophile-04 approved these changes Feb 6, 2025

View reviewed changes

rin-st approved these changes Feb 6, 2025

View reviewed changes

technophile-04 mentioned this pull request Feb 8, 2025

Add yarn fleek command to deploy on fleek #1029

Closed

technophile-04 added 2 commits February 11, 2025 13:32

Merge branch 'main' into ipfs-bgipfs

d65efbb

remove scripts dir

000cf04

technophile-04 approved these changes Feb 11, 2025

View reviewed changes

azf20 and others added 2 commits February 19, 2025 11:15

bump bgipfs

c537986

Merge branch 'main' into ipfs-bgipfs

4da1e68

fix yarn lock

c22af20

Pabl0cks approved these changes Feb 21, 2025

View reviewed changes

Pabl0cks merged commit b1da2ca into scaffold-eth:main Feb 21, 2025
1 check passed

Pabl0cks mentioned this pull request Feb 21, 2025

Update node version requirements on Readme #1052

Merged

technophile-04 mentioned this pull request Feb 24, 2025

backmerge 24-02-25 scaffold-eth/create-eth#218

Merged

github-actions bot mentioned this pull request Feb 24, 2025

Version Packages scaffold-eth/create-eth#219

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use bgipfs for upload #1039

Use bgipfs for upload #1039

azf20 commented Jan 25, 2025

technophile-04 left a comment

azf20 commented Jan 26, 2025

portdeveloper commented Jan 27, 2025

technophile-04 commented Jan 27, 2025

azf20 commented Jan 28, 2025

Pabl0cks left a comment •

edited

Loading

azf20 commented Feb 3, 2025

portdeveloper commented Feb 4, 2025

carletex commented Feb 4, 2025

technophile-04 left a comment

rin-st left a comment •

edited

Loading

technophile-04 left a comment •

edited

Loading

carletex commented Feb 11, 2025

Pabl0cks commented Feb 11, 2025

azf20 commented Feb 19, 2025

Pabl0cks left a comment

carletex commented Feb 21, 2025

Use bgipfs for upload #1039

Use bgipfs for upload #1039

Conversation

azf20 commented Jan 25, 2025

Description

technophile-04 left a comment

Choose a reason for hiding this comment

azf20 commented Jan 26, 2025

portdeveloper commented Jan 27, 2025

technophile-04 commented Jan 27, 2025

azf20 commented Jan 28, 2025

Pabl0cks left a comment • edited Loading

Choose a reason for hiding this comment

azf20 commented Feb 3, 2025

portdeveloper commented Feb 4, 2025

carletex commented Feb 4, 2025

technophile-04 left a comment

Choose a reason for hiding this comment

rin-st left a comment • edited Loading

Choose a reason for hiding this comment

technophile-04 left a comment • edited Loading

Choose a reason for hiding this comment

carletex commented Feb 11, 2025

Pabl0cks commented Feb 11, 2025

azf20 commented Feb 19, 2025

Pabl0cks left a comment

Choose a reason for hiding this comment

carletex commented Feb 21, 2025

Pabl0cks left a comment •

edited

Loading

rin-st left a comment •

edited

Loading

technophile-04 left a comment •

edited

Loading