I see you iterate over all triangles for every tile, which is not efficient at all.

I see two options:

Iterate all triangles once and rasterize them to the framebuffer, just like the HW or traditional software rasterizers do it it.
To support parallelism, the usual practice (as demonstrated by Dreams or UE5) is to use 64 bit atomic min operations.
The depth goes into the high bits, so atomic min does a depth test. The remaining bits are use to store triangle data. E.g. instance id, triangle id, barycentric coords.
After that we have a visibility buffer, and we need to shade it in a second pass, see http://filmicworlds.com/blog/visibility-buffer-rendering-with-material-graphs/

The other approach is tile based, but requires an acceleration structure. To reduce it's size, ideally we work with clusters of triangles, not single triangles.
For each tile we build a frustum and traverse the AS in loose front to back order to find intersecting clusters.
We raster the cluster to a tile framebuffer in LDS memory, so no need for slow atomics to global VRAM.
Because we do in front to back order, we can also implement occlusion culling to quickly reject clusters or whole branches of the AS.

The second approach is very interesting, but also very complex and i think it has been never done yet.
I might try it in case i decide to use point splatting instead triangles.
With triangles the complexity is quite high, and i'm not sure we really need insane high tessellation as demonstrated by UE5 Nanite. They can do it, but i guess games will end up using lower geoemtry resolution in practice. And if so, no need for a software rasterizer.

Anyway, please share your results, as it's always interesting… ; )

AhmedSaleh said:
Do you have an optimization or pseudo code to overcome that ? or at least for option 1 ?

No. The problem isn't that simple. If you simply want to iterate pixels and find stuff to cover it, the efficient solution is raytracing, requiring an acceleration structure too.

We could for example bin the triangles to tiles (basically a low res framebuffer). So when processing tiles you only rasterize triangles which are actually visible in them very likely.
But such binning isn't trivial: We don't know how many triangles go into a tile, and one triangle goes into an unknown number of tiles. So we get a problem which would require a form of dynamic allocation like std::vector, which can't be done efficiently.

Well, people often think raytracing is more complex than rasterization, but it's the opposite. Especially if we want parallelism.
If you want efficiency, complexity will grow a lot, causing something like 10 x more code than you have now or more. : /

I remember a ‘new’ approach of rasterization which was introduced with simd CPUs. Set up 3 edge equations, make bounding rectangle over the triangle, and for each pixel evaluate all 3 equations in parallel ‘for free’ due to simd instructions.
That's similar to what you do (as i understand), and much simpler than the traditional scanline approach used in early 3D games. It also seems more parallel at first.
But interestingly, looking at Nanite source code, they came back to the scanline approach afaict. One thread rasterizes one triangle using that. This means high runtime divergence across threads, minimized only by the fact that all their software triangles are guaranteed to be small.
One workgroup rasterizes one cluster of triangles, and there are background threads performing a hacky and imperfect but simple and good enough approach of occlusion culling using the reprojected depth of the previous frame.
Those background threads are persistent, and implement a multi producer / multi consumer queue on queue on GPU, which is not even specified to work by APIs, but they found it does work an any HW they tried.
That's how they get parallelism (background threads produce cluster to draw, rasterizer threads consume them), which is quite adventurous, but also quite complex.
There is some Nanite Deep Dive video on YT wroth to watch. Dreams also had presentation and paper, really interesting:

hmmm… following that Epic example, one idea would be to bin or sort the triangles by pixel area. So triangles nearby in the sorted list have a similar count of pixels. That's one dispatch for the binning or sorting.

Then a second dispatch would process the triangles from the sorted list. So all threads in a workgroup raster one triangle, and they'll have similar count of pixels, so all threads finish at nearly the same time and saturation is good.
Would work with both the scanline or the edge equations approach (for the latter you would sort by are of the bounding rectangle, not the triangle). Not sure what's better. The latter wastes half of work on processing pixels which are not inside the triangle, the former has more code divergence due to complexity. Might not matter too much.

That's very simple and not that bad. But the downside is that nearby threads will write to different locations in the framebuffer. But maybe due to the atomic access this does not make it so much slower than it is anyway.

Though, i guess for just a single cow model it might not beat your current approach. The scene should be more complex to make it a win.

AhmedSaleh said:
Can you please go to basics and explain what's the problem with my approach

If i get it right, you iterate over all triangles for every pixel. (I'm mainly confused about the line ‘if (x < vbosize && y < vbosize)’, so not sure)
That's inefficient. Think of the other way around: For each triangle, process the whole screen and test each pixel to be in or outside. That's the same amount of work, but maybe the inefficiency becomes more obvious to see. It's brute force.

Ideally we want to process and triangle only once, and each pixel only once. That would be very efficient, but not possible.
Usually on GPU we aim to achieve efficiency on a coarse level to limit the work.
On the fine level (usually within a single workgroup), brute force is often the best option, due to the parallel hardware.
But on top of that we have the same requirements to be efficient as usual, e.g. when working with a single core, because GPUs have not an infinite number of cores.

Likely you have observed this already yourself, and it brought you to the idea to use tiles to increase efficiency.
And by rephrasing it, i can only help to make you more aware, or to look at it from a general perspective first.
Which then may help to come up with an algorithm for your specific case more easily, or at less uncertainty along the way.

AhmedSaleh said:
and what's the approach in github is doing ? possibly with pictures if you can

Well, they already try to do this. Giving an overview and some pictures. It does not clarify all the details, but i won't study their code just to summarize their work better.

One interesting point i catch from looking at their summary is the limitation of 1024 triangles per tile. So that's their way to address (or dodge) the dynamic allocation problem i have mentioned.
But they admit it's not a general solution. It works only if the application can guarantee to keep within the limit. That's a very common solution for games, because a general solution is not possible within our tight performance budgets.

The scanline approach is probably general, but no idea how it works.

However, looking at their FPS numbers for single low poly models, neither approach seems practical or worth the effort. Why not just use the HW rasterization pipeline, which has more features, is orders of magnitudes faster and no work needed to be done at all?
What's your motivation to work on this?

For learning purposes, anything is fine, as you'll learn some thing from any approach.

If you want something practical for games, solving problems where HW rasterization fails, Nanite might be the better resource to look up. And investment of serious time and effort is expected.
Personally i think software solutions do make sense, but only if we solve some more problems than just rasterization along our way, like occlusion culling and LOD.
Notice those latter problems are still open, while simple rasterization is not. Rasterization is solved, even so well that we have HW acceleration for decades.