In the engine I'm working with, I'm drawing some polygons to be used as a stencil for a portal of sorts, and I want to clip depth to it, in the sense that anything I draw after setting up the stencil, won't draw anything closer than the polygons I drew to the stencil buffer. I want to do this to prevent some odd clipping issues.

This is my code thus far.

void RENDER_StartStencilFromPoly(vertexStruct* v, int n)
{
	glEnable(GL_STENCIL_TEST);

	glStencilMask(0xFF);
	glClearStencil(0);
	glClear(GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
	glStencilFunc(GL_NEVER, 1, 0xFF);
	glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);
	
	// Draw the poly
	RENDER_SetPolyFlags(POLY_ZBUFFER);
	glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

	for (int i = 0; i < n; i++)
	{
		Vertices[i].position.x = (v[i].x / 65536.0f) / 256.0f;
		Vertices[i].position.y = (v[i].y / 65536.0f) / 256.0f;
		Vertices[i].position.z = (v[i].z / 65536.0f) / 256.0f;
	}

	RENDER_SetArrays(ARRAY_MAIN);
	glDrawArrays(GL_TRIANGLE_FAN, 0, n);

	// Restore settings.
	glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

	glStencilMask(0x00); // Disable writing. Now prepared to draw within our stencil.
	glStencilFunc(GL_EQUAL, 1, 0xFF);
	glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
}

The stencil works, but once I'm done drawing, stuff that's closer than the polygon I used for the stencil, bleeds through stuff I draw after ending the stencil. I want to clamp things to the inside of this "portal". I've tried many mixes of StencilFuncs and StencilOps, but I can't seem to produce the effect I'm looking for.

Here's a video of the issue in question: https://shadowmavericks.com/files/ShareX/tfury_2024-06-07_00-44-01.mp4

The draw order is as follows: Turn on stencil and draw polygon (function above) -> draw contents of tunnel -> turn off stencil -> draw transparent cap over tunnel.

I've also heard about clipping planes, but have absolutely no clue how to use them in my context, since most tutorials are cryptic as hell and are overly verbose with the mathematical terminology. If clipping planes would help me in my situation, could someone explain how I could use it in this context?

Found a solution. What I had was mostly correct, but I needed to flush the ZBuffer, then re-draw my portal triangles to the depth buffer after drawing the contents of the portal.

void RENDER_StartPolyStencil(vertexStruct* v, int n)
{
	glEnable(GL_STENCIL_TEST);
	glDepthMask(GL_TRUE);

	glStencilMask(0xFF);
	glStencilFunc(GL_NEVER, 1, 0xFF);
	glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);

	glClearStencil(0);
	glClear(GL_STENCIL_BUFFER_BIT);
	
	// Draw the stencil polygons
	RENDER_SetPolyFlags(POLY_NOBIND); // Handles glEnable/Disable stuff.
	glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

	for (int i = 0; i < n; i++)
	{
		Vertices[i].position.x = (v[i].x / 65536.0f) / 256.0f;
		Vertices[i].position.y = (v[i].y / 65536.0f) / 256.0f;
		Vertices[i].position.z = (v[i].z / 65536.0f) / 256.0f;
	}

	RENDER_SetArrays(ARRAY_MAIN);
	glDrawArrays(GL_TRIANGLE_FAN, 0, n);

	// Restore settings.
	glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

	glStencilMask(0x00); // Disable writing. Now prepared to draw within our stencil.
	glStencilFunc(GL_EQUAL, 1, 0xFF);
	glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
}

Then, draw your portal contents.

Then, call this:

void RENDER_EndPolyStencil(vertexStruct* v, int n)
{
	glDepthMask(GL_TRUE);
	glDepthFunc(GL_ALWAYS); // This will let me overwrite depth.
	
	// Re-draw the stencil polygons with some depth bias to the z-buffer.
	RENDER_SetPolyFlags(POLY_ZBUFFER | POLY_DEPTH_BIAS_FAR | POLY_NOBIND); // Handles glEnable/Disable stuff.
	glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

	for (int i = 0; i < n; i++)
	{
		Vertices[i].position.x = (v[i].x / 65536.0f) / 256.0f;
		Vertices[i].position.y = (v[i].y / 65536.0f) / 256.0f;
		Vertices[i].position.z = (v[i].z / 65536.0f) / 256.0f;
	}

	RENDER_SetArrays(ARRAY_MAIN);
	glDrawArrays(GL_TRIANGLE_FAN, 0, n);

	// Re-enable the color buffer.
	glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
	glDepthFunc(GL_LEQUAL); // Restore depth func to <=.
	glDisable(GL_STENCIL_TEST);
}