I’ve been working on the z/VM TLS Server – that worker machine that handles data-in-flight encryption to and from the host layer of IBM z/VM – on and off for about twenty years now.
I can say with some confidence that no one has ever accused it of having a simple and elegant design. This isn’t by choice; TLS, certificate management, and encryption in general are beasts when it comes to its requirements, and an elegant means of configuration is no mean trick, especially in that Venn diagram of “security” and “mainframe.”

All the same, the idea that it has too few buttons was new to me. The DTCPARMS file, which controls z/VM TCP/IP server config, is “non-trivial” in its scope. And yet – and yet! – there’s always room for another option, and we’ve uncovered lately that we may have too few knobs present. For example, there is no button for:

  • granular control of key sizes
  • control of elliptic curves
  • adjustment of the default order of ciphers

If you, dear reader, have a need for such, do open a Requirement against IBM. But that’ll take a while, and I have a mitigation in the interim.

Introducing TCPRUNXT

TCPRUNXT.SAMPEXEC is shipped with z/VM, and it’s a Rexx-scripted exec for TCP/IP configuration exits. It is automatically called – you don’t need to enable it, it’s there when you need it, always! Usually, we hope you don’t need it, but in today’s world, it comes up more often than usual. It will reside on one of TCPMAINT.591 disk as a sample; copy it over to the TCPMAINT.198 disk for local configuration (don’t mess with the IBM file; always keep your mods away from where we do updates).

The good news is that this entry point providdes a basis for modding the TLS server in a way that meets security policy! The bad news is that you have to write a bit of code. The good news is that I’m going to give you some samples. Here we go!

How to Modify TCPRUNXT

First, make that local copy. Then, XEDIT the file. Around line 250, there will be a space for your local modifications. You’re going to insert a line that says:

When (calltype = "BEGIN") & (parms = _ClassSSL) Then

This logic says, “when the server’s coming up, & if it’s an SSL machine, do this stuff.” You’ll then procede to make modifications here. What kind of mods, you ask? Let me explain.

System SSL on z/VM

We talk a lot about the TLS server on z/VM, but we don’t often expose the fact that z/OS System SSL has been ported and is running under the covers. This is the set of modules which undergoes FIPS 140-n evaluations; it’s vital to crypto operations for host connectivity. But we usually don’t mention interfaces in – why would we? The TLS server machine is there to do that work for us.

cough except when there’s no knob for that cough

All the same, System SSL has a boatload of environment variables (PDF) for your use. Normally, this is handled for you. Today, we’re going to use TCPRUNXT to set a few. Just remember that you’ll need to reboot your TLS servers (SSLADMIN STOP and SSLADMIN START) before changes take effect.

Let’s get on with a few example, though!

Sample code: reordering ciphers

/*------------------------------------------------------------*/ 
/* For SSL class servers, ensure the GSK_V3_CIPHER_SPECS      */ 
/* variable is reordered to meet cipher order preferences.    */ 
/*------------------------------------------------------------*/

When (calltype = "BEGIN") & (parms = _ClassSSL) Then          
  Do                                                          
    "GLOBALV SELECT CENV" ,                                   
       "SETL GSK_V3_CIPHER_SPECS_EXPANDED ",                  
       "00380035002F003900330032C001"                         
    Say "Environment variable GSK_V3_CIPHER_SPECS_EXPANDED ", 
       "set to: 00380035002F003900330032C001"                 
  End

To be clear, this is a garbage example of cipher codes – only seven. You’re going to want to compare this versus what’s in the z/VM TCP/IP Planning & Customization Guide. Most folks don’t have an issue with the current default cipher list (this is the RFC standard approach we use), but if you need to hijack the ordering because a client application got creative and artistic with its approach to standards implementation: we can help.

Sample code: Modified ECURVE list

/*------------------------------------------------------------*/ 
/* For SSL class servers, remove NIST-P192 and NIST-P224      */ 
/* curves by modifying GSK_SERVER_ALLOWED_KEX_ECURVES. See    */
/* Table 29 in the z/OS System SSL doc for which curve has    */
/* what code.                                                 */ 
/*------------------------------------------------------------*/

When (calltype = "BEGIN") & (parms = _ClassSSL) Then          
  Do                                                          
    ecurve_list = ‘002500240023’
    "GLOBALV SELECT CENV" ,                                   
       "SETL GSK_SERVER_ALLOWED_KEX_ECURVES ",                  
	       ecurve_list
    Say ”Environment Variable GSK_SERVER_ALLOWED_KEX_ECURVES modified “,
	      “to “ ecurve_list “.”
  End

There is no option today for stripping ecurves. FIPS mode will turn off the brainpool curves, but our only other options just mess with cipher codes. If you have a particular need to drop a curve – e.g., you’re looking ahead at Common Criteria compliance to the NIAP Virtualization Protection Profile or similar – then this is the hammer you want to swing.

EDIT: GSK_SERVER_ALLOWED_KEX_ECURVES is for server-side processing. GSK_CLIENT_ECURVE_LIST is for client-side connections. You may wish to use both, depending upon what you’re planning to do.

Sample code: Extended Master Secrets

/*------------------------------------------------------------*/ 
/* For SSL class servers, enable Extended Master Secret       */
/* support through use of GSK_CLIENT_EXTENDED_MASTER_SECRET.  */
/* This support is only used by TLS 1.0 through TLS 1.2 and   */
/* will not be applicable to future TLS 1.3 support.          */
/*------------------------------------------------------------*/ 
 
When (calltype = "BEGIN") & (parms = _ClassSSL) Then         
 Do                                                         
    "GLOBALV SELECT CENV" ,                                  
       "SETL GSK_CLIENT_EXTENDED_MASTER_SECRET ",              
/*                                                            */ 
/* --select ON or REQUIRED based on local security policy.    */ 
           "GSK_CLIENT_EXTENDED_MASTER_SECRET_ON"
/*         "GSK_CLIENT_EXTENDED_MASTER_SECRET_REQUIRED"       */
/*                                                            */
     Say "Environment Variable GSK_CLIENT_EXTENDED_MASTER_SECRET ",
           "has been set to ON."
 End

EMS is a TLS 1.2-specific option, and won’t matter when z/VM does finally implement TLS 1.3, but for FIPS compliance in the modern age, you may need to turn this on. Do double-check what your security policy is and uncomment certain lines accordingly.

Sample code: Ephemeral DH Group Sizes

/*------------------------------------------------------------*/ 
/* For SSL class servers, establish group size via the GSK_   */ 
/* _EPHEMERAL_DH_GROUP_SIZE variables, for policy compliance  */ 
/*------------------------------------------------------------*/

When (calltype = "BEGIN") & (parms = _ClassSSL) Then          
  Do                                                          
    group_size = '2048’
    "GLOBALV SELECT CENV" ,                                   
       "SETL GSK_CLIENT_EPHEMERAL_DH_GROUP_SIZE ",                  
	        group_size     
    "GLOBALV SELECT CENV" ,                                   
       "SETL GSK_SERVER_EPHEMERAL_DH_GROUP_SIZE ",                  
	        group_size                                             
    Say ”DH Group Size “ group_size “established for DHE via “,
	        “environment variables GSK..DH_GROUP_SIZE”    
  End

This one is setting multiple variables in a single swing, based on a variable one can modify the next time there’s a rule change.

Closing thoughts

First: the code is sample-only and is provided as-is. You’re going to want to test this out before you push things out to prod.

Second: based on internal testing, this only works for the z/VM TLS Server and not the z/VM LDAP Server. LDAP has other interfaces for specifying environment variables (as a server also ported from z/OS), and there aren’t many ways around its current limitations.

Third: this does not absolve IBM of providing a reasonable (and supported!) means of making these changes.

Fourth: test in test, deploy to prod, and use Rexx Say statements to remind you of where you are and what you’re doing.

The z/OS System SSL documentation (linked above) is vital for understanding what options you have, and they will have tables upon tables upon tables that you can consult. Not everything z/OS does is applicable to z/VM, though, so take care in making changes if you’re not already in consultation with IBM Support.

This does come up often enough, though, that I gave an impromptu talk on it at the 2025 VM Workshop, and if the movers and shakers in our space are tripping over this, there’s a good chance you may as well. I hope this has helped! Do drop me a line if a link breaks or if you have any suggestions or questions.

-bwh.