Description
GSCR (Grad Special Committee Request) is a Cold Fusion application running on CStack, and is owned by Custom Dev (Shannon Osbourne's group). As of time of writing, the application is in production on webhost032, with dev/test on webhost031. Because this app was bequeathed multiple times, nobody remembers how it was set up, how to maintain it, or what to do when something goes wrong.
The application broke after upgrading Cold Fusion, and patches were reverted until a solution could be found. Due to an undiagnosed bug in either Cold Fusion or Java, jar files containing libraries needed for the app's proper execution may be loaded out of order. Because we do not have a support contract with Adobe, they would not take a bug report on this issue, but they did acknowledge Due to the exceptionally cluttered method of java deployment used, whereby multiple versions of libraries are included in various jar files, the application could not properly load required specific versions, and would crash. Typical symptoms in the Cold Fusion exception log point to NoClassDefFoundError , sometimes regarding DescriptionFactory or HttpClient .
Redeploying the application on a new VM worked initially. The new prod machine is webhost106. However, redeploying dev/test in exactly the same fashion to webhost107 failed. No reason has been found why the jars load out of order.
Ultimately, the correct solution was to update axis2 and install the jaxws-ri libraries as well, then customize (i.e., delete) sub-libraries that were known to be buggy or causing additional conflicts.
The following filesystem changes are not affected by puppet.
The customer is responsible for modifying application files, including:
Solution
In summary:
- Move ksb-client-1.0.3-jar-with-dependencies.jar so that it gets loaded last
- Update axis2 (https://axis.apache.org/axis2/java/core/) - used version 1.7.4
- Install jaxws-ri (https://jax-ws.java.net/) - version 2.2.10
- Change Cold Fusion jvm arguments to load jaxws-ri/ libraries as additional classpath.
- Delete unneeded or conflicting axis2 libraries
- Delete unneeded or conflicting jaxws-ri libraries
- Update App Config Files
- Test
Move ksb-client
The client supplied file ksb-client-1.0.3-jar-with-dependencies.jar contains conflicting libraries. To avoid versioning issues, this needs to be loaded last. The last directory to be loaded in the CF Server Class Plath is:
/infra/lamp/mounts/gscr/gscr-test/jars/
Update axis2
Obtain axis2 libraries from https://axis.apache.org/axis2/java/core/ :
http://www-us.apache.org/dist/axis/axis2/java/core/1.7.4/axis2-1.7.4-bin.zip
MD5: 1b7f70d424f9e2733fe944bdc01f90c7
From /www/{instance-name}/server10/lib:
mv axis2/ axis2~
unzip axis2-1.7.4-bin.zip
mv axis2-1.7.4/lib/ axis2/
chown -R lamp:root axis2/
cd axis2
mv axis2-jaxws-1.7.4.jar axis2-jaxws-1.7.4.jar~
mv axis2-saaj-1.7.4.jar axis2-saaj-1.7.4.jar~
mv jaxb-impl-2.2.6.jar jaxb-impl-2.2.6.jar~
mv jaxb-xjc-2.2.6.jar jaxb-xjc-2.2.6.jar~
The directories in /www/gscr-test/server10/lib/ also need to be moved out of the way. Make sure the following are no longer available:
jaxb-api.jar
jaxb-libs.jar
jaxb-xjc.jar
jax-qname.jar
jaxrpc.jar
Update axis.jar
The original axis library is no longer available for download from apache. Instead, use apache maven repository:
http://central.maven.org/maven2/axis/axis/1.4/axis-1.4.jar
MD5: 03dcfdd88502505cc5a805a128bfdd8d
From /www/{instance-name}/server10/lib:
mv axis.jar axis.jar~
wget -O axis.jar http://central.maven.org/maven2/axis/axis/1.4/axis-1.4.jar
axis.jar does not need to be unpacked.
Install jaxws-ri
Reference implementation of the Java API for XML Web Services (JAX-WS) specification.
http://repo.maven.apache.org/maven2/com/sun/xml/ws/jaxws-ri/2.2.10/jaxws-ri-2.2.10.zip
MD5: 2231396f77c2680ffd047f9c7b190036
unzip jaxws-ri-2.2.10.zip
cd /www/gscr-test/server10/lib/
cp -r ~/jaxws-ri/lib/ jaxws-ri
chown -R lamp:lampgrp jaxws-ri/
The jaxws-rt.jar library is buggy when used in this app. Disable or remove it.
mv jaxws-ri/jaxws-rt.jar jaxws-ri/jaxws-rt.jar~
Several of the sub-libraries in the jaxws-ri distribution replace jars in the main CF lib/ directory. Remove them.
mv jaxb-api.jar jaxb-api.jar~
mv jaxb-libs.jar jaxb-libs.jar~
mv jaxb-xjc.jar jaxb-xjc.jar~
mv jax-qname.jar jax-qname.jar~
mv jaxrpc.jar jaxrpc.jar~
Reconfigure ClassPath
Modify jvm.config to include the jaxws-ri/ directory and its contents.
vi /www/gscr-test/server10/bin/jvm.config
-Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/lib/jaxws-ri,{application.home}/gateway/lib/ .... etc.
Update App Config Files
GSC has to have client-sign.properties file in the htdocs directory, e.g. /www/gscr-test/htdocs/client-sign.properties
That file must have a correct information about Cynergy connection: username, password, and location of the keystore file.
The correspondent keystore file should be in this directory: /www/gscr-test/confs/cynergy-test.keystore
The reset of the configuration information is located in the appliacation gsc.ini file which is located in htdocs/gscr/gsc.ini
Testing / Verification
Navigating to the URL https://gsc-test.hosting.cornell.edu/test_permits.cfm?reset should report something like the following. If it doesn't work, a CF error will be thrown.
local.urlAddress=http://localhost:80/CUWAPortal/Permit?netid=efk3@cornell.edu&permits=all
efk3@cornell.edu Users EN-Subdelegates-gs CIT-CF.cornelledu OIT-PGP_dist OIT-lamp.humphreys.upd-gs OIT-lamp.actuate.upd-gs CIT-Subdelegates-gs CIT-LAMP-GS EN-ENGFS-HRDATA-AP-gs EN-ENGFS-HRDATA-gs EN-DelegateGroups-gs ga.DISTRIBUTION.ateam cit.coldfusion.support cit.is.coea.user.aux.prod cit.is.inf cit.itso.sec-council cit.itso.seclist cit.lamp.support cit.nocdocs cit.roc cit.staff cit.systems cit.systems.sfam cu.employee cu.student ece.techshop ece.vpn.faculty ece.vpn.staff ece.vpn.student ece.vpn.techshop engr.as.emprep engr.eas gr.gradreps.prod gr.gradreps.test mae.vpn oit-subdelegates-gs ga.SENDAS.webservices ga.DISTRIBUTION.webservices OIT-IDM-ARSSubDelegates-gs OIT-IDM-ARSSubDelegates-ls OIT-lamp.entomology.upd-gs rg.president.employee.staff rg.president.employee rg.cuniv.employee rg.cuniv.employee.staff rg.finaff.employee rg.finaff.employee.staff CIT-CF-GS rg.cuniv.student rg.cuniv.student.undergrad rg.ce.student rg.as.student rg.ce.student.undergrad rg.as.student.undergrad EN-EC-Admins CIT-VSSMediaspaceCUnivEmployees en-cc-vpn-staff CIT-Radius-eduroam SF-EN-ADMGRP SF-EN-GEN-ADMGRP SF-EN-PUB-ADMGRP SF-EN-RES-ADMGRP SF-EN-UTL-ADMGRP EN-CI-AllUnitsAllUsers rg.cio.employee rg.cio.employee.staff rg.cit-inf.employee.staff rg.cit-inf.employee CIT-box-enterprise-users test-dynjv CIT-webex-enterprise-users rg.cio-svc.employee rg.cio-svc.employee.staff en-cs-vpn-acad en-ec-vpn-acad en-ec-vpn-pd83 EN-CC-Security EN-EC-Staff CIT-Static.r3symposium CIT-skillsoft-enterprise-users EN-EC-Amplab-Logon ga.DISTRIBUTION.coecis-logs ga.SENDAS.coecis-logs ga.ADMIN.coecis-logs ga.ACCESS.coecis-logs en-ec-vpn-staff EN-ec-git CIT-workday-testers LIB-EZProxyTestGrp CIT-allow-twofactor en-ec-argon-users LIB-EZProxy-active-us ga.DISTRIBUTION.cit-sf-support ga.DISTRIBUTION.cit-syseng CIT-AT-eligible-fac-staff-gradst LIB-EZProxy-student-us LIB-EZProxy-employee-us CIT-SharePoint-Users CIT-IDM-CUEmployeeWithoutRetiree CIT-optin-twofactor CIT-Static.uPortal.Cornell CIT-Static.ombudsman CIT-Static.IsSecurity CIT-Static.infra CIT-force-twofactor cit.csm.access CIT-IAVE-AV-DOWNLOADS CIT-Static.channel7 CIT-Static.idstatictest CIT-Static.cit CIT-Static.selfservice.admissions CIT-Static.activity CIT-Static.sandbox CIT-280534934474-poweruser
200 OK done!
Another URL, https://gsc-test.hosting.cornell.edu/pingCynergy.cfm , should return something like:
/www/gscr-test/htdocs/gscr/
/www/gscr-test/htdocs/gscr/
cynergy-test
========
This comes from the service
DV=DISB;ND=DISB;AD=DEPS;PREQ=DISB;CM=DISB;CCR=CCRD;CTRL=DEPS;PRNC=MLBK;CMNC=MLBK