Check malloc() return value on client->server ClientCutText message. Client can send up to 2**32-1 bytes of text, and such a large allocation is likely to fail in case of high memory pressure. This would in a server crash (write at address 0).

libvncserver/rfbserver.c

@@ -2461,6 +2461,11 @@ rfbProcessClientNormalMessage(rfbClientPtr cl)
 	msg.cct.length = Swap32IfLE(msg.cct.length);
 
 	str = (char *)malloc(msg.cct.length);
+	if (str == NULL) {
+		rfbLogPerror("rfbProcessClientNormalMessage: not enough memory");
+		rfbCloseClient(cl);
+		return;
+	}
 
 	if ((n = rfbReadExact(cl, str, msg.cct.length)) <= 0) {
 	    if (n != 0)