BlockifyVR (Virtual Reality) #1732

Brackets-Coder · 2024-10-23T21:51:23Z

BlockifyVR

This is a Virtual Reality extension I've been working on since January 27th of 2023.

The end goal is to have:

Little to no performance overhead with the extension
Cross-platform compatibility for all major 6DOF headsets A-frame supports
An easy-to-use system that's coherent and efficient
Works well with any 3D engine
Bug-free enough for optimal user experience without frequent disruptions

Here are some things to note:

It includes a local copy of the A-frame JS library, version 1.6.0, MIT License. This is for VR Management, rendering, and ease of development as I'd have to write the entire thing from scratch using WebXR, WebGL, and the Gamepad API otherwise. Setting up the VR boilerplate and stereoscopy and other things would be a pain without it.
Gallery thumbnail, documentation, and sample projects are not currently available and will be worked on as I get nearer towards release
The rendering uses a plane fixed to the camera which directly mirrors the scratch stage with an unshaded texture each frame. This is relatively performant and works well, but there could be better solutions to look into. The plane is placed 1 meter away from the camera and A-frame uses stereoscopic rendering to reduce eye strain. Note that the plane automatically scales to the size of the camera (display) while attempting to maintain the ratio of the stage. I use this instead of drawing directly to the canvas because it won't let me and I would have to re-write the renderer that's already tailored for VR'.
The above note means that ONLY THE STAGE LAYERS will be rendered. Any HTML Elements above the stage canvas (reporter monitors, "ask" block, etc.) WILL NOT work in a VR session.
Sorry the code's a bit messy and outdated and bad

Upcoming release plan:

~~v1.0-pre-alpha: Earliest version. Minimal functionality, proof-of-concept version. Not available here. Not open source.~~

~~v1.0-alpha: work-in-progress. Expect significant bugs and poor performance. Should not be used.~~

~~v1.0-beta: Cross-platform compatibility, major bug fixes, and optimizations have been added.~~

v1.0-release-candidate: Current version. Feature complete. Meets standards of full release with minimal issues. During this release period, preparations for full release include gallery thumbnail, sample project, documentation, etc.

v1.0: First release. Feature complete and meets all standards of great performance, very few issues, cross-platform compatibility, and intuitive design.

v1.5: Bugfixes and optimizations. Possibly a small feature or change, such as controller vibrations.

v2.0: Big update. Most likely something like hand tracking support for some platforms. Also will include some minor bugfixes and optimizations.

v2.2: Bugfixes and optimizations.

Active Development

v1.0-release-candidate

Bug fixes and optimizations
Prep for release
Sample project
Documentation

hammouda101010 · 2024-10-28T12:35:47Z

that looks cool

hammouda101010 · 2024-10-29T12:26:53Z

that looks cool

It looks cool on the flat screen right now, but it's not so great when you try it. You'd probably get motion sickness. I'm still working on it.

sadly i dont think that's possible to stop this.

Xeltalliv · 2024-10-29T13:55:46Z

Does this show the same scratch 3D perspective to both eyes?

Brackets-Coder · 2024-10-29T13:58:57Z

Does this show the same scratch 3D perspective to both eyes?

A-frame has stereoscopic rendering built in and I've enabled it in the rendering settings, so no. This fixes some issues with visualizing depth, so the eye offset allows your brain to process the image with more clarity. Even if the matrices aren't used in the renderer, it should properly use the correct eye offset. Why do you ask, just out of curiosity? If you'd like I could add a block that disables or enables this feature, in case the Scratcher wants to build that framework themselves, which I'd doubt.

Keep in mind this is still a work in progress and most of the code is many months old so it's not the most efficient way to do VR.

Xeltalliv · 2024-10-29T14:17:58Z

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

Brackets-Coder · 2024-10-29T14:32:33Z

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

I believe that there can be two XRViewerPose matrices for each eye if available, according to Mozilla Docs. You might want to look into that again. It'd be great if Augmented Reality could support AR headsets such as the Oculus Quest 3 in addition to just mobile touchscreen devices. There should be a views array that can do this. An example can be seen here. Just a suggestion though. I simply think having AR headsets would be a nice update.

Xeltalliv · 2024-10-30T01:20:09Z

I'll take a look when I have time. Thought it will probably be hard because the only kind of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

Brackets-Coder · 2024-10-30T01:29:36Z

I'll take a look when I have time. Thought it will probably be hard because the only kinf of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

As for VR testing, you wouldn't really need a VR headset. I've used Meta Quest's Immersive Web Emulator extension for testing when I didn't care to load up the entire headset.

Drago-Cuven · 2024-10-30T10:28:19Z

ah yes, I remember seeing a trailer for this on scratch. I wanted to help, but now I can't

Drago-Cuven · 2024-10-30T12:36:37Z

nvm I think I can
i'm Martinelplayz on scratch btw

Thebloxers998 · 2024-11-01T08:18:33Z

could i try the extension?

Hi siskka7 it's me Thebloxers998

siskka7 · 2024-11-01T08:40:57Z

Thebloxers998

hmm https://github.com/TurboWarp/extensions/blob/2837a7adfbfc2e3aba49108003686affa1cb7beb/extensions/MasterMath/BlockifyVR.js

Thebloxers998 · 2024-11-01T13:46:16Z

Thebloxers998

hmm https://github.com/TurboWarp/extensions/blob/2837a7adfbfc2e3aba49108003686affa1cb7beb/extensions/MasterMath/BlockifyVR.js

I'm not sure what you're "hmm"ing. It's just the source code.

Hi, nice extension

Xeltalliv · 2024-11-02T22:24:40Z

I'll take a look when I have time.

I took a look and have no idea. It's all code specific to AFRAME. I don't know AFRAME. AFRAME is too big and complicated for me to try to debug, especially if the only tool in my disposal is WebXR API Emulator browser extension.

Thebloxers998 · 2024-11-02T22:51:05Z

Nic3

Brackets-Coder · 2025-01-15T14:38:08Z

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.

Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

siskka7 · 2025-01-15T15:26:06Z

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.

Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

Brackets-Coder · 2025-01-15T17:55:49Z

I just published a new release version here which should fix the issue with the "when button" blocks. Note that I haven't yet tested this to confirm it's working but it was working last time before I changed the inputs to accept reporters. I've reverted this change.

Note that this release also brings experimental WORK IN PROGRESS changes to the "get matrix" block which SHOULD NOT be used.

For now, the best alternative is to just use the default perspective projection Simple3D offers and use the "camera FOV" block with it. This will only work if you confirm your headset rotation transformations are computed in the right order (ZXY, with headset Y rotation as X and headset X rotation as Y in Simple3D, varies between renderers)

siskka7 · 2025-01-15T18:14:12Z

here is my test project that logs vr input https://github.com/siskka7/siskka7/blob/main/vr%20blockify%20input%20logger.sb3
(tutorial in project) note that this is before was "when block" was fixed i hope this will help developing

Drago-Cuven · 2025-01-15T18:33:36Z

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

https://github.com/immersive-web/body-tracking

siskka7 · 2025-01-15T19:16:55Z

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)
probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.
Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

I believe that the "When button pressed" block issue you're describing is due to a recent change where I made the block inputs allow reporters, but I don't think hat blocks like inputs that allow reporters. I'll get another release out soon that should fix this issue. I'm not sure what you mean by the other blocks though, everything else seems to work fine except the get matrix block

i think there is big misunderstanding i dont thought hat block but (picture1)

and the menu looks like (picture2)

siskka7 · 2025-01-15T19:24:14Z

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)
probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.
Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

I believe that the "When button pressed" block issue you're describing is due to a recent change where I made the block inputs allow reporters, but I don't think hat blocks like inputs that allow reporters. I'll get another release out soon that should fix this issue. I'm not sure what you mean by the other blocks though, everything else seems to work fine except the get matrix block

i think there is big misunderstanding i dont thought hat block but (picture1) and the menu looks like (picture2)

intriguing. Turbowarp menus aren't supposed to look like that, but I can guarantee it's probably not an extension issue.

so there is no fix idea :(

siskka7 · 2025-01-16T06:23:35Z

Don't know why the "when button" blocks still aren't working... sigh I guess I have another bug to fix cuz it was working earlier
Oh did not mean that

Thebloxers998 · 2025-01-16T20:48:37Z

Don't know why the "when button" blocks still aren't working... sigh I guess I have another bug to fix cuz it was working earlier

Dang, but how did it end up bugging. Did you check the logic of the block

Brackets-Coder · 2025-01-21T00:38:53Z

Progress Update

So turns out looking at the same 1700 lines of code for a year messes with your brain and makes you have dumb oversights that waste your time.

The "when button pressed" issue wasn't actually there at all (the extension code was fine), but rather it was a unique, project-specific backwards compatibility issue with scratchblocks that resulted in my demo project after changing the acceptReporters property of the input. All I had to do was delete the block and switch it out with the updated one.

I also think I fixed the matrix processing issue. I was previously using a A-frame component to attempt to get the pose from each frame and the xrReferenceSpace, but A-frame doesn't have any way to return the referenceSpace it's using internally - all the matrices I needed were stored in the camera component. I've removed the custom A-frame component, as it was redundant and useless. This provided an extra performance boost :)
For the most part, the matrix block is returning the proper values for the A-frame camera matrices, but for some reason I can't get it to work with Simple3D? It looks really weird and just plugging the camera FOV block into the default start with [perspective] projection block works perfectly for some reason... I'm assuming this is because Simple3D (webGL) uses -Z and so do the matrices which cause the renderer to use +Z and result in this? Maybe I should try VR extension matrix blocks with the AR extension's sample project...

@Xeltalliv since you originally wrote the getMatrix block it'd be great if you could take a look just to verify. I've heavily modified it but it still serves the same purpose. I don't know if I did the combined matrix right but the other ones seem to work fine. Also don't know if I'm handling NaNs correctly either.

the image is from my computer but when you use VR mode it looks really weird

The values of these blocks match the names of their respective matrix arrays (except combined)
          matrix: {
            acceptReporters: false,
            items: [
              {
                text: Scratch.translate("combined"),
                value: "combined", 
              },
              {
                text: Scratch.translate("projection"),
                value: "projectionMatrix",
              },
              {
                text: Scratch.translate("view"),
                value: "matrixWorld",
              },
              {
                text: Scratch.translate("inverse view"),
                value: "matrixWorldInverse",
              },
            ],
          },

    getMatrix({ matrix, item }) {
      if (item < 1 || item > 16) return "";

      let camera = ACamera.components.camera.camera;
      if (matrix == "combined") {
        if (camera.projectionMatrix.elements.some(isNaN) || camera.inverseViewMatrix.elements.some(isNaN)) return 0;
        return combinedMatrix.identity().multiplyMatrices(camera.projectionMatrix, camera.inverseViewMatrix).elements[item - 1] || 0;
      }
      return camera[matrix]?.elements[item - 1] || 0
    }

I should have a new commit soon.

Brackets-Coder · 2025-01-21T00:49:18Z

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)
probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.
Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

I believe that the "When button pressed" block issue you're describing is due to a recent change where I made the block inputs allow reporters, but I don't think hat blocks like inputs that allow reporters. I'll get another release out soon that should fix this issue. I'm not sure what you mean by the other blocks though, everything else seems to work fine except the get matrix block

i think there is big misunderstanding i dont thought hat block but (picture1) and the menu looks like (picture2)

@siskka7 you'd be pleased to know I fixed this issue too.. turns out when updating Scratch.translate() all of the values of the device menu where headset even though the text was different...

I think I need more sleep.

• Removed redundant pose-matrices component • Re-wrote all of get matrix block • Added new camera FOV block • fixed issues with some blocks & other minor block changes

Brackets-Coder · 2025-01-21T01:35:17Z

Also probably a Sample Project soon
Definitely going to open up for review again soon
I'll also continue on working on the website, documentation... etc. along with maybe some extra surprises (no promises)

v1.0-release-candidate-4 is available here (latest release & commit above)

Brackets-Coder · 2025-01-21T20:17:14Z

Everything in the entire extension is now working as expected except for combined matrices.

Projection and view matrices work as expected.

Brackets-Coder · 2025-01-22T02:09:36Z

just confirmed it also works in packaged HTML files

Brackets-Coder · 2025-01-24T01:55:27Z

forgot to mention that sample project is now available for testing at https://brackets-coder.github.io/BlockifyVR/project.html

Brackets-Coder · 2025-01-24T02:42:48Z

Just fixed combined matrices. Turns out the matrix name was matrixWorldInverse instead of inverseViewMatrix... these little oversights are the ones that get me.

Working on adding menu button support for oculus-touch-controls then it's basically done

Brackets-Coder · 2025-01-24T20:54:11Z

I guess this is ready for review now?

Thebloxers998 · 2025-01-24T21:54:16Z

I guess this is ready for review now?

Yayyyyyyyy!!!!!!!

Thebloxers998 · 2025-01-24T21:54:45Z

I guess this is ready for review now?

Yayyyyyyyy!!!!!!!

I Happy Now :)

Thebloxers998

You have done some nice progress!

Brackets-Coder · 2025-01-24T21:59:19Z

You have done some nice progress!

I guess but there could be more it doesn't have controller vibrations and the menu button on Oculus touch controls isn't supported but other than that it's basically done I guess

Brackets-Coder · 2025-01-25T21:18:02Z

So....

A-frame doesn't support detecting the menu button/pause button or the Oculus/system button on the oculus-touch-controls component. I tried detecting these manually through the Gamepad API but I guess they're not exposed or something because they don't work and I checked with ChatGPT and it said you couldn't get them.

I'm leaving the "menu" and "system" buttons in the menus of the blocks because they should still work on vive and windows mixed reality

Which speaking of I need people who can test non-Oculus platforms because I haven't really tested them at all.

Other than that I think this extension should be ready for release. I know a lot of them are busy but it'd be great if we could get some reviewers/moderators over here... this extension is moderately large so I expect it to take longer than normal to review

Thebloxers998 · 2025-01-26T12:19:20Z

So....

A-frame doesn't support detecting the menu button/pause button or the Oculus/system button on the oculus-touch-controls component. I tried detecting these manually through the Gamepad API but I guess they're not exposed or something because they don't work and I checked with ChatGPT and it said you couldn't get them.

I'm leaving the "menu" and "system" buttons in the menus of the blocks because they should still work on vive and windows mixed reality

Which speaking of I need people who can test non-Oculus platforms because I haven't really tested them at all.

Other than that I think this extension should be ready for release. I know a lot of them are busy but it'd be great if we could get some reviewers/moderators over here... this extension is moderately large so I expect it to take longer than normal to review

Ok

Thebloxers998

You've done some nice changes 👍

Brackets-Coder · 2025-01-29T00:33:13Z

I think I'm going to make a few changes to make the code more "professional" and less messy but I can't right now so I'll get around to it when I'm less busy with other things

Brackets-Coder added 5 commits October 23, 2024 17:17

Create BlockifyVR.js

2837a7a

Update extensions.json

6d7d9a6

Run format

fa9a88d

Fix validate

e35cb13

Fix ESLint

9f822b1

Brackets-Coder marked this pull request as ready for review October 24, 2024 19:02

Brackets-Coder marked this pull request as draft October 24, 2024 19:02

Brackets-Coder changed the title ~~BlockifyVR (Virtual Reality) Extension~~ New Extension: BlockifyVR (Virtual Reality) Oct 24, 2024

Brackets-Coder changed the title ~~New Extension: BlockifyVR (Virtual Reality)~~ BlockifyVR (Virtual Reality) Oct 24, 2024

SharkPool-SP added the pr: new extension Pull requests that add a new extension label Oct 24, 2024

Merge branch 'TurboWarp:master' into blockify-vr

16a676d

This comment was marked as resolved.

Sign in to view

bug fixes, matrix updates, new blocks, etc.

16a89d9

• Removed redundant pose-matrices component • Re-wrote all of get matrix block • Added new camera FOV block • fixed issues with some blocks & other minor block changes

Brackets-Coder marked this pull request as ready for review January 21, 2025 14:50

Thebloxers998 reviewed Jan 24, 2025

View reviewed changes

Update BlockifyVR.js

6bd1fa5

Thebloxers998 reviewed Jan 26, 2025

View reviewed changes

Merge branch 'TurboWarp:master' into blockify-vr

1363a97

BlockifyVR (Virtual Reality) #1732

Are you sure you want to change the base?

BlockifyVR (Virtual Reality) #1732

Conversation

Brackets-Coder commented Oct 23, 2024 • edited Loading

BlockifyVR

Upcoming release plan:

Active Development

hammouda101010 commented Oct 28, 2024

hammouda101010 commented Oct 29, 2024

Xeltalliv commented Oct 29, 2024

Brackets-Coder commented Oct 29, 2024 • edited Loading

Xeltalliv commented Oct 29, 2024

Brackets-Coder commented Oct 29, 2024 • edited Loading

This comment was marked as resolved.

Xeltalliv commented Oct 30, 2024 • edited Loading

This comment was marked as resolved.

Brackets-Coder commented Oct 30, 2024

Drago-Cuven commented Oct 30, 2024

Drago-Cuven commented Oct 30, 2024

This comment was marked as resolved.

Thebloxers998 commented Nov 1, 2024

siskka7 commented Nov 1, 2024

Thebloxers998 commented Nov 1, 2024

Xeltalliv commented Nov 2, 2024

This comment was marked as resolved.

Thebloxers998 commented Nov 2, 2024

Brackets-Coder commented Jan 15, 2025

siskka7 commented Jan 15, 2025

This comment was marked as resolved.

Brackets-Coder commented Jan 15, 2025

siskka7 commented Jan 15, 2025 • edited Loading

Drago-Cuven commented Jan 15, 2025

siskka7 commented Jan 15, 2025

siskka7 commented Jan 15, 2025

siskka7 commented Jan 16, 2025

Thebloxers998 commented Jan 16, 2025

Brackets-Coder commented Jan 21, 2025 • edited Loading

Progress Update

Brackets-Coder commented Jan 21, 2025

Brackets-Coder commented Jan 21, 2025 • edited Loading

Brackets-Coder commented Jan 21, 2025

Brackets-Coder commented Jan 22, 2025

Brackets-Coder commented Jan 24, 2025

Brackets-Coder commented Jan 24, 2025

Brackets-Coder commented Jan 24, 2025

Thebloxers998 commented Jan 24, 2025

Thebloxers998 commented Jan 24, 2025

Thebloxers998 left a comment

Choose a reason for hiding this comment

Brackets-Coder commented Jan 24, 2025 • edited Loading

Brackets-Coder commented Jan 25, 2025

Thebloxers998 commented Jan 26, 2025

Thebloxers998 left a comment

Choose a reason for hiding this comment

Brackets-Coder commented Jan 29, 2025

Brackets-Coder commented Oct 23, 2024 •

edited

Loading

Brackets-Coder commented Oct 29, 2024 •

edited

Loading

Brackets-Coder commented Oct 29, 2024 •

edited

Loading

Xeltalliv commented Oct 30, 2024 •

edited

Loading

siskka7 commented Jan 15, 2025 •

edited

Loading

Brackets-Coder commented Jan 21, 2025 •

edited

Loading

Brackets-Coder commented Jan 21, 2025 •

edited

Loading

Brackets-Coder commented Jan 24, 2025 •

edited

Loading