Bad Gateway error when running reports from Edit Mode

[csizemore]

we been round the block a million times with IBM on this issue with no luck.  we are running CA 11.1.7 FP2 in a mixed environment (windows and linux).  When we run a report from edit mode, we get a 504 Bad Gateway error.  report runs fine if run normally, just getting the error when running from edit mode.  we've tried from the loadbalanced URL, directly from the GW URI's and directly from the dispatcher URI's with no luck.  we've run and pulled .har and logs from chrome that gave nothing and the Cognos logs are relatively unhelpful.

the only lines that i can pull from session logs that show any issue are below (IP addresses removed):

2021-08-03T13:54:09.279-0400   WARN   Trace.dispatcher   [Default Executor-thread-83922]   4jlGd8hM2qjC2C8GM888Glywqq4y2ChvlvG9vsdl   jCCMlvGy8wjqq8ysyGhyGhGMCM288l8l82lCjyw4   vGs99s4w2syqMqGGvqw88C4vl8M42lMGMqhsMyC2   670859c7-8988-4888-b78c-166ec09d34f2   ###.###.###.###   54856      AggAAADjvBJhAAAAAAoAAAB/r/KJ2ggj1oXlFAAAAIAh5i/E0SM0gaDIMQLwA0XdRGg8BwAAAFNIQS0yNTYgAAAAZb9fM6E5Dz0U4WzHvAvEIk5Xo7x4WNHSunS2vQfvRLA=      IBM Cognos   4312 ###.###.###:9301   4312   2021-08-03 13:54:09.279   -4   0B473C4656131EAED3892AA66DF2A4FF6F418FC72B66DAA54D6F0AB3F4BB1ECC_{_670859c7-8988-4888-b78c-166ec09d34f2_}_   q4vM49shCwqld4Gdhj2dj29dy8dwCwlwqsjGhCsG   1      Default Executor-thread-83922   DISP   7   3   Trace.dispatcher      [test_do_html_errors] com.cognos.pogo.handlers.fault.TestDoHTMLErrorPage   TRACE   

2021-08-03T13:54:09.279-0400   DEBUG   com.cognos.pogo.pdk.BasicHandler   [Default Executor-thread-83922]   4jlGd8hM2qjC2C8GM888Glywqq4y2ChvlvG9vsdl   jCCMlvGy8wjqq8ysyGhyGhGMCM288l8l82lCjyw4   vGs99s4w2syqMqGGvqw88C4vl8M42lMGMqhsMyC2   670859c7-8988-4888-b78c-166ec09d34f2   ###.###.###.###   54856      AggAAADjvBJhAAAAAAoAAAB/r/KJ2ggj1oXlFAAAAIAh5i/E0SM0gaDIMQLwA0XdRGg8BwAAAFNIQS0yNTYgAAAAZb9fM6E5Dz0U4WzHvAvEIk5Xo7x4WNHSunS2vQfvRLA=      IBM Cognos   4312   invoking [test_do_html_errors] com.cognos.pogo.handlers.fault.TestDoHTMLErrorPage

2021-08-03T13:54:09.279-0400   DEBUG   com.cognos.pogo.handlers.fault.TestDoHTMLErrorPage   [Default Executor-thread-83922]   4jlGd8hM2qjC2C8GM888Glywqq4y2ChvlvG9vsdl   jCCMlvGy8wjqq8ysyGhyGhGMCM288l8l82lCjyw4   vGs99s4w2syqMqGGvqw88C4vl8M42lMGMqhsMyC2   670859c7-8988-4888-b78c-166ec09d34f2   ###.###.###.###    54856      AggAAADjvBJhAAAAAAoAAAB/r/KJ2ggj1oXlFAAAAIAh5i/E0SM0gaDIMQLwA0XdRGg8BwAAAFNIQS0yNTYgAAAAZb9fM6E5Dz0U4WzHvAvEIk5Xo7x4WNHSunS2vQfvRLA=      IBM Cognos   4312   do HTML error page? - false

environment build:
Loadbalanced external URL round robin'ing to Cognos GWs.
Linux Cognos GW's, using httpd.
Linux Dispatchers
Windows CMs.