Posts on The IT Stories

Notes from the field: Omnissa Workspace ONE UEM e-mail based enrollment OG

Tue, 31 Dec 2024 16:51:45 +0000

When configuring Omnissa UEM you seem to be able only to select the top OG in a SaaS environment when selecting e-mail based enrollment, this is called auto discovery, see https://docs.omnissa.com/bundle/WorkspaceONE-UEM-Managing-DevicesV2306/page/ConfigureEnrollmentOptions.html when completing the FTU for e-mail based enrollment afterwards you can go in and select the OG again an then drill down in the desired OG. Pretty straightforward.

Hope it helps.

Notes from the field: Workspace ONE UEM custom attribute assignment rule limitations

Tue, 31 Dec 2024 14:15:42 +0000

When bulk enrolling pre-existing devices or auto-pilot devices you can use a custom rule / attribute assignment on e.g. a serial number to move the corresponding devices to a deeper OG which is preferred. Only in a situation with 1200+ devices you might encounter DB maximum issues in SaaS and need to contact support. This was the case for me and my customer and sadly no solution regarding this movement rule, it seems like a hard maximum and conflicts would occur on the database in question. When using Freestyle Orchestrator with the advanced licens this would not be an issue and you would be able to fix this issue before it can occur, for us the workaround was allowing a top OG enrollment for the devices and manually bulk move to the corresponding OG.

Notes from the field: Apple DEP devices not correctly installing Workspace ONE Intelligent HUB

Tue, 31 Dec 2024 14:07:11 +0000

When encountering a failed Apple MacOS device enrolment from the DEP program and using Workspace ONE UEM, it might be that there is an bug related to the intelligent hub from a deployment perspective. This was the case for my customer after a bulk enrolment of new devices out of the blue would be having issues, this was later confirmed in https://kb.omnissa.com/s/article/6000198 the solution for this in a future scenario would be to flip the released version of the hub to a last known state from a company/user perspective and when resolved flip it back to the most current version. This appears to only work for the MacOS side of things. iPhone etc. use the public app store and latest version which can’t be controlled this way.

Notes from the field: Workspace ONE UEM, Apple Federation and the APNS account

Tue, 31 Dec 2024 14:01:02 +0000

Most companies I encounter don’t have a clear understanding of apple accounts… Well an apple account is personal and it’s not of the company even if the domain in question is being used for personal apple accounts. How can you change this? Well the company domain is from the company and then you can claim it for federation authentication. See https://support.apple.com/guide/apple-business-manager/intro-to-federated-authentication-axmb19317543/web aftewards there will be a grace period of 30 days before a temporary apple-id account is assigned and the logon needs to be done and migrate it to a uniquely e-mail/apple-id account that is not from the company/business, it’s personal after all.

Notes from the field: Workspace ONE Access SAML Signing with 3rd party certificate

Tue, 31 Dec 2024 13:50:26 +0000

On a recent customer deployment we got the requirement of that all certificate signing would be signed from a 3rd party trusted certificate provider. This is all no problem and you can follow this: https://docs.omnissa.com/bundle/workspace-one-access-administration-guide/page/GenerateandUseanExternalSigningCertificateforSAMLAuthenticationinWorkspaceONEAccess.html but keep the following in mind:

Existing signing certificates and an import is not possible
The request and signing needs to be done from Access, importing is not possible of an already validated certificate
Keep the 1 year maximum in mind and with renewals and SAML SP/IDP configurations keep in mind that an dynamic import can take some time

Hope it helps.

Notes from the field: Workspace ONE UEM iOS/iPhone model smart groups

Tue, 31 Dec 2024 13:39:38 +0000

Just a quick blog regarding Apple device classification for iPhone/iPad, you might be a bit hesitant in using this regarding the “legacy” filter being stamped upon it. This is for now as it is and everything will be supported when this will be fully moved to the new OEM & Model filte options that now are not classified for Apple devices. Support confirmed this and all is well again.

Hope it helps.

Notes from the field: Workspace ONE UEM, Invites, OG and language

Tue, 31 Dec 2024 13:34:28 +0000

When configuring an OG structure and customising templates for e.g. device enrolment invites you might encounter an issue that the expected language is not updating. The solution for this is changing it on the top OG in question: groups and setting » all settings » organisation group» details » Locale or achieve this with an override on the one you would like it to apply.

Hope it helps.

Notes from the field: Horizon First-Gen / Next-Gen migration

Tue, 31 Dec 2024 13:30:17 +0000

After all the updates and changes around company structure and licensing it’s finally there the EOL of First-Gen control plane and customers should migrate to the Next-Gen control plane. This all sounds easy enough but at my customer who was still using the First-Gen control plane for licensing only the CSP logon that should present the Next-Gen control plane did not provide this. For this we opened up a support case and after some troubleshooting this resulted in a hard backend V1 option that needed to be switched to V2 and afterwards we would be able to logon to the First-Gen and Next-Gen control planes.

Notes from the field: VMware/Broadcom/Omnissa CSP connector changes

Tue, 31 Dec 2024 13:24:58 +0000

Earlier this year my customer would get an CSP migration e-mail regarding the connector based deployment scenario would be deprecated moving forward with VMware/Broadcom and the latter acquiring VMware. This is regarding https://docs.vmware.com/en/VMware-Cloud-services/services/setting-up-enterprise-federation-cloud-services/GUID-76FAECB3-CFAA-461E-B9C9-2A49C39CD17F.html

After some discussion and support case feedback around this and explaining that connector less isn’t a valid option for our use case with Workspace ONE (now Omnissa) and many question marks around that one it finally became clear it’s two seperate things now and the solution would be:

Notes from the field: Citrix NetScaler VLAN tagging and Hyper-V / VMM

Tue, 31 Dec 2024 13:06:06 +0000

Long story short if you want to use VLAN trunk tagging, Hyper-V itself will not let you see this in the GUI and this is only supported via CLI/Powershell and further down the road VMM will allow this in an compute fabric for GEN2 only! (and NetScaler is still GEN1) see https://charbelnemnom.com/what-is-vlan-trunk-mode-in-hyper-v-hyperv/ and https://learn.microsoft.com/en-us/system-center/vmm/vm-settings?view=sc-vmm-2025&tabs=AddvNIC%2CConfigureQoS%2CProcessorThrottling#support-for-trunk-mode

After this would be configured via CLI/Powershell the entire VLAN tagging is out of the VM and configured on the network adapter itself, I’ve had some discussion with Citrix around GEN2 support and roadmap but sadly no new updates on that.

Notes from the field: Citrix NetScaler "VPX" hard drive errors

Tue, 31 Dec 2024 12:51:40 +0000

In a particular case with one of our customers we encountered the reporting messages “Hard disk drive errors” in our logging and checked the solution with the customer, at first all seemed to be alright and no issues but this particular node would keep giving the event entries. From article https://support.citrix.com/s/article/CTX214458-netscaler-vpx-hard-drive-errors?language=en_US all indicates that the underlying hypervisor and this particular VM would be problematic. We’ve decided to rip the HA setup and remove the faulty node and re-add it with a fresh deployment, afterwards no error messages anymore, the storage itself for all the nodes encountered a previous failover and put in some corruption on this specific VM (others we’re save as to no issues or corruption).

Notes from the field: The case of AppLayering and Office updates or is it?

Tue, 31 Dec 2024 12:37:05 +0000

Earlier this year we had an customer who out of the blue have Office disappearing and all issues regarding native/published application dependancy hell with it. This would be a recurring issue each time around vacation time of the customer. Well we work with a peal the onion principle and started out dissecting the image of the customer how it was build, in this case with AppLayering and nicely separated layers for the customers of the customer and so on. It looked like a bug in the o365 layer but eventually this wasn’t a bug with the layers at all. After some long troubleshooting also with Citrix support the issue was in the Office 365 portal and that Office would be updated even with ignoregpo settings in place to disable all update functionality. The bothersome option in Office365 was the cloud update option which in turn ignores and re-updates the ignore gpo setting and forceful patching all versions of office. Disablement of this would resolve the issue.

Notes from the field: Citrix StoreFront cannot change password New UI (Technical Preview)

Tue, 31 Dec 2024 12:17:47 +0000

In a recent troubleshoot with a customer it came to light that password change message would occur and a broken form would be shown to the user. After some troubleshooting it came to light that the next gen UI would be enabled and password change options aren’t fully feature parity yet. The at logon would allow you to change but not always work, and after logging in and looking for a change password option that wouldn’t be there. All the other options in StoreFront itself would be correct.

Notes from the field: Citrix NetScaler Native OTP stopped working

Tue, 31 Dec 2024 12:07:59 +0000

In a recente P1 outage at a customer we got messages that some users would not be able to logon and some would, authentication debugging would show cascading events that would work and some that would deny logons. In this particular setup there are two MFA solutions, one not native to the NetScaler and one native using the parameter setup in ADDS and the tokens. The latter would be the one failing, customer already did a reboot and failover of the nodes with no avail. After checking with all the engineers of the customer and one goosy Mick ;-) we came to the conclusion there was a time drift on the NetScaler, it would start out as 30 seconds behind, and moving up to about 5 minutes and authentications would fail. For us and this particular case the reboots wouldn’t work and re-entering the NTP setup also wouldn’t help, the latest 13.1 release which resolved a CVE also resolved our problem.

Notes from the field: Citrix CVAD site and the year destruction bug

Tue, 31 Dec 2024 12:01:12 +0000

Just for the holiday we got a strange issue that from the CVAD side and MCS updates the interactions wouldn’t work anymore and that machines would stay on and no power cycle actions would be working throughout the night for the daily reboots. After a troubleshoot with the customer we found out that one of the hypervisor hosts (Hyper-V) got an update in the middle of the night and flipped to the future as of 2025 and within the hour reverted itself. The entire flip caused that the DB was in a filthy state and never recovered to the state of 2024 the current date and time. Opened up a support case for the issue and not many helpful insights there so we decided to move further and recover from a database and snapshot revert (veeam) to a working previous state. This all resolved the issue but also triggered the missed scheduled reboots as after the restore (so the maintenance schedule for the weekend was a good point for us and the customer)

Notes from the field: Citrix AppLayering 2409 Edge changes

Tue, 31 Dec 2024 11:51:52 +0000

In the latest upgrade for a customer who was using application layers for their browser got a nice empty browser after the latest upgrade to Citrix AppLayering. Turns out that starting from version 2409 the only supported behaviour is an OS layer for that. See the following https://support.citrix.com/s/article/CTX399422-how-to-configure-microsoft-edge-in-an-app-layering-environment?language=en_US

This is fundamental design change that you should be aware of, there is an option to put in an RFE for some other path but that shouldn’t be the preferred way imho.

Notes from the field: Citrix NetScaler and the authentication policy limit

Tue, 31 Dec 2024 11:44:15 +0000

For a customer who is using the NetScaler as a IDP/SP service with the NetScaler Gateway and some service provider requirements we had a nice policy limitations which caught us by surprise: [https://support.citrix.com/s/article/CTX227301-error-32-authentication-policies-are-already-bound-while-binding-authentication-policy?language=en_US#:~:text=When%20multiple%20policies%20(two%2Dfactor,binding%20for%20one%20virtual%20server.](https://support.citrix.com/s/article/CTX227301-error-32-authentication-policies-are-already-bound-while-binding-authentication-policy?language=en_US#:~:text=When%20multiple%20policies%20(two%2Dfactor,binding%20for%20one%20virtual%20server.)

The mentioned RFE is still in the works or forgotten but to solve this little puzzle you’ll need to go to the 31st policy binding or earlier and create a policy label with all the other authentication policies that you would want to go beyond 32 and replace the latter in the original set with a NO_AUTHN policy and the newly label as next factor.

Notes from the field: Citrix NetScaler Admin Partitions Cleanup

Tue, 31 Dec 2024 11:34:55 +0000

In a previous blogpost we’ve discussed some pain points regarding a whole bunch of admin partitions and the inability to upgrade regarding disk space usage, see it here: https://technicalfellow.com/2023/12/notes-from-the-field-citrix-netscaler-partitions-performance-and-pain/

After a support case and discussions about when to use the reporting feature of the NetScaler or not, we needed a solution to clean all those old .pdb files.

In our particular case we had the 13.1 release of the NetScaler and the option for expanding the disk wasn’t available, have to check if this is still needed or not, but our solution was the following:

Notes from the field: Citrix NetScaler, partitions, performance, and pain

Fri, 22 Dec 2023 19:29:18 +0000

On a recent joint project with my partner in crime Anton van Pelt there was a long outstanding support issue which needed our dedicated attention. Long story short we have a customer in a nice new greenfield which got migrated from F5 to NetScaler and the introduction of Citrix Gateway and migrated the backend to admin partitions in favor of requirements stating so. To a maximum we would see 50/70 partitions for that setup.

Notes from the field: Citrix Gateway DTLS fail-over UDP/TCP

Tue, 07 Nov 2023 17:28:50 +0000

On a recent troubleshoot a customer complained that after a failover all ICA sessions would do a fallback to TCP and not uplift again to UDP until the DTLS checkmark would be disabled/enabled

Well this worked in the past but since 13.0 build 58.x we have the ability to create a DTLS listener VIP for this, and all the custom items it has with it - Configure DTLS VPN virtual server using SSL VPN virtual server (netscaler.com)

Notes from the field: Horizon and the locked.properties debacle

Wed, 25 Oct 2023 16:56:20 +0000

On a recent Horizon deployment version 2212.1 we just couldn’t get the workings correctly with the portalhost/balancedhost entries, the “workaround” for that matter until we get it sorted out is to turn back on the unexpected host feature like below:

allowUnexpectedHost=true checkOrigin=false enableCORS=false

Well this worked only for the original installed URL of the connection servers and load-balancer for that matter, all other SAN entries which were valid and should be used we’re providing the error: Error Pager failed to load

Notes from the field: Just one of those days that nothing goes as expected

Wed, 25 Oct 2023 16:45:13 +0000

Just a quick blog that these day’s it can be very difficult or very simple in regards of troubleshooting and resolving issues in a deployment. Let me take you on a journey ;)

Journey 1:

UAG deployment fresh as fresh and no working logon page after a valid deployment, at first troubleshooting, troubleshooting, redeployed with the customer and still the same result, validated across versions, vSphere, UAG etc. and to no effect.

Notes from the field: Citrix Gateway Advanced Auth RADIUS SSO not working

Fri, 15 Sep 2023 20:39:08 +0000

In the last couple of months, I came across some NetScaler redeployments regarding the latest CVE and on the other hand moving over from basic policies to advanced policies.

For this to work we are depending on the AAA setup with authentication profiles to combine it all with a Citrix Gateway deployment.

For RADIUS there is a nice article containing a how to: Deployment Guide: Learn how to configure Citrix Gateway to use nFactor to authenticate against a RADIUS server for MFA

Notes from the field: The Kerberos chronicles, the one with certificate-based authentication

Wed, 31 May 2023 20:03:37 +0000

If you’ve read my previous Kerberos chronicles blogs you see a trend with the Microsoft patches, hardening updates and with this one the upcoming strong mapping / full enforcement mode of certificate-based authentication. See the following article for explanation: KB5014754—Certificate-based authentication changes on Windows domain controllers - Microsoft Support

This one is going to have a big impact if left unchecked and doesn’t get the proper attention it needs.

An example is being described here: Certificate-Based Authentication Changes and Always On VPN | Richard M. Hicks Consulting, Inc. (richardhicks.com)

Notes from the lab: VMware Workspace ONE and the home-lab setup for one external IP

Mon, 24 Apr 2023 19:45:18 +0000

Just a quick blog for setting up your home-lab and use all the VMware Workspace ONE services on the UAG’s with one external IP.

Our starting point is based on the following articles:

Unified Access Gateway Appliances Deployed in a Double DMZ (vmware.com) – follow the steps for double DMZ deployment, Minimum/Optional Horizon Protocols and if needed switch the ports to be used for BEAT, take note that UDP 443 is by default reserved on the UAG see Solved: Can BEAT run over a different port than UDP 8443? - VMware Technology Network VMTN

Notes from the lab: Some magic, integrating Citrix resources with VMware Access

Wed, 01 Mar 2023 16:14:48 +0000

Like my friend Edwin de Bruin explains in his blog(s): Migrating from Citrix Gateway to VMware Access Workspace One: Part one (debruinonline.net) and Migrating from Citrix Gateway to VMware Access Workspace One: Part Two! (debruinonline.net) he expects me to deliver you all some magic.

For this blog I’m going to start with the necessary resource articles and blogs for a starting point and those are the following:

Providing Access to Citrix-Published Resources in VMware Workspace ONE Access – this one is the basic starting point for integration and should help you out with a working setup

Notes from the field: Microsoft Azure MFA Number Matching and the one with NPS extension

Mon, 06 Feb 2023 15:34:31 +0000

Regarding the upcoming change of Microsoft MFA number matching, some customers started to ask me hey what’s going on? Do we need to do something? Is there any impact for our users?

Well, the short answer is yes.

The long answer is well it depends, can we live with the current setup or is there something going to break?

I tested this out in my lab setup in which I have this configured for some scenarios like Citrix Gateway, VMware UAG/Connection Servers etc.

Notes from the lab: Citrix XenMobile 10.15 upgrade fails

Wed, 11 Jan 2023 13:23:27 +0000

Regarding my own XenMobile deployment I had a 10.14 Rolling Patch environment 6 running and updated it to Rolling Patch 9 before the eventual upgrade to 10.15 base. This all is very easy to do and all the required information is presented at Release notes for Rolling Patches | XenMobile Server Current Release (citrix.com) regarding patches, requirements etc.

Well first back-up of the database and an offline XenMobile appliance snapshot so that I could always revert and afterwards upgraded to the latest 10.15. The appliance did a reboot a then everything broken see below:

Notes from the field: Citrix NetScaler Azure subscription-based licensing

Wed, 11 Jan 2023 13:07:53 +0000

Just a quick blog regarding a deployment model of Citrix NetScaler on Azure. There is an option to use subscription-based licensing for a deployment, meaning you pay by the hour it is running in Azure. See Deploy a Citrix ADC VPX instance on Microsoft Azure for more details.

This setup was chosen by a customer of mine which didn’t have any new licenses and wanted to migrate to such a model. This all worked fine but later the customer wanted to upgrade the license/throughput model of the deployment. Well after a quick discussion on the Citrix slack and support we came to the disappointing response this is not possible. The only way to upgrade is a redeploy of the instance with such a license model and restore of back-up or in case of an HA deployment break HA and redeploy it in that way.

Notes from the field: VMware UEM Apple Registered mode pre-registration not working

Thu, 29 Dec 2022 20:37:16 +0000

On a recent project in which we use VMware UEM in a managed and registered mode of management the latter is experiencing a bug when pre-registering for enrollment is enabled.

With closed enrollment enabled all devices need to be registered beforehand, this is a form of security to mandate there is no open enrollment possible. This all works fine except for Apple devices, when enrolling the device in an managed state and it’s not registered at all the enrollment would pass the first payload for MDM by Apple and then get a you are not allowed to enroll, this imho is a step to late because you are already communicating and allowing sort of access in the environment.

Notes from the field: The Kerberos chronicles, the one with Citrix NetScaler

Thu, 29 Dec 2022 17:52:14 +0000

The same as my previous Kerberos blog but this time we have Citrix NetScaler in the mix with drumrolls… Kerberos Constrained Delegation henceforth to be known as KCD.

This in an setup derived from the following article: Tutorial: Azure Active Directory single sign-on integration with Citrix ADC SAML Connector for Azure AD (Kerberos-based authentication) - Microsoft Entra | Microsoft Learn

The scenario is a working setup from one of my customers and regarding all the latest exploits we needed to upgrade the NetScaler versions to a higher and higher and higher version. Along the way we encountered a faulty KCD which never would work again. Only a downgrade to the working version firmware would resolve the issue for us.

Notes from the field: The Kerberos chronicles, the one with VMware TrueSSO

Thu, 29 Dec 2022 14:19:39 +0000

After a lengthy and cumbersome troubleshoot on a VMware TrueSSO setup finally had the time to blog this one.

In summary the situation with a customer was a working VMware TrueSSO setup which stopped working, after lengthy troubleshooting we opened a support case with VMware and later also with Microsoft.

The issue was manifesting in an always denied by policy module message form the CA which also cohosts the enrollment server role of VMware TrueSSO. Logging etc. nothing was making any sense and had multiple sessions with VMware and Microsoft around this. We already had some suspicions that CIS hardening around this was perhaps breaking items in the chain and eventually it was the culprit.

Notes from the field: The broken VMware Access, UEM, and HUB portal

Sun, 20 Nov 2022 12:13:33 +0000

On a recent project with a customer, we encountered the issue that the VMware integration of the three products would be sort of “broken”. We first observed the issue after implementing the Intelligent Hub Verify rule set and see that this wouldn’t work. The devices and UEM wouldn’t show in the portal and the access policy would never apply. This in turn was caused by having the wrong OG configured in VMware Access, this was a weird issue because the entire setup was configured that way months earlier and was validated to work.

Notes from the field: VMware App Volumes LDAP(S) lockout

Sat, 19 Nov 2022 19:42:01 +0000

This is a quick blog to address a lockout issue if you are having troubles with LDAP(S) and or the validation of the certificate. When you want to validate this or for that matter resolve it because you can’t login to the App Volumes Manager anymore do the following on the database:

Select the dbo.ldap_domains entry and click the Select Top 1000 Rows to view the entries

Edit Top 200 Rows to edit the value in question and flip it to your value

Notes from the field: VMware Access CRL url too long?

Mon, 31 Oct 2022 14:02:42 +0000

This is just a quick post regarding CRL checking in VMware Access. It seems that when you have the “NEW” UI interface enabled there is a bug when you put in a valid CRL location in the lengths of: http://this.ismycrlfilelocation.crl that it would chop the end off and stay at http://this.ismycrlfilelocation and then a faulty CRL location.

Solution is switching it over to the “OLD” UI interface and put the complete URL in and switch back.

Notes from the lab: Using VMware Access as IdP for Citrix Gateway

Sun, 02 Oct 2022 14:51:57 +0000

I like to fiddle around with possibilities when it comes to SAML, OAUTH authentications. This all started when a customer engineer triggered me with the possibility of achieving an SSO experience with the Citrix NetScaler and using VMware Access as the source of truth for authentication.

Well guess what this works! And even for the native workspace app users as well. All with the conditional access policies of VMware Access before it.

Notes from the field: Citrix Cloud Connector we’re having trouble signing you in

Mon, 19 Sep 2022 12:17:50 +0000

Recently I’ve encountered an issue after installing the Citrix Cloud Connector on new Windows Server 2022 machines. The configuration on my first machine went just fine until the sign-in, here my interest got peeked because I still have IE11 on my box, strange that it uses the sign-in for Citrix Cloud.

Well let’s test the second one and remove IE11 from the box and see what happens:

Oh boy, so we are depending on IE11. Installed it again and voila:

Notes from the field: The one that Android said no more local

Sun, 11 Sep 2022 18:58:16 +0000

On one of my projects, we’ve encountered a strange issue regarding domain name resolving.

A little background on the canvas painted it’s about a VMware Workspace ONE setup with working web URL’s and UEM enrollments, you name it. We have a nice setup regarding managed devices and these use a per-app VMware tunnel connection to achieve secure application access. All working dandy fine. The devices are based on a Ascom Myco 3 setup with Android 10 and are dedicated used for specific applications.

Notes from the lab: Migrating Azure AD Connect and then we cannot sync

Sun, 04 Sep 2022 18:06:39 +0000

This is a quick blog post regarding my own Azure AD Connect migration and a nasty error after trying to connect again for an initial connection and synchronisation.

A little insight in my environment, I already had the latest version running of Azure AD Connect namely 2.1.16.0 on my Windows Server 2019. See Azure AD Connect: Version release history - Microsoft Entra | Microsoft Docs

So, I spun up a new Windows Server 2022 and installed the Azure AD Connect role on it, imported my configuration file like described here How to import and export Azure AD Connect configuration settings - Microsoft Entra | Microsoft Docs

Notes from the field: Citrix XenMobile / CEM don't touch that store name!

Sat, 13 Aug 2022 18:30:43 +0000

Just a quick shout out blog to stress the importance of the store name that XenMobile / CEM uses. This default store name is called “Store”.

If you by any means have changed this store name to anything else, you might run in two issues depending on different scenarios.

Scenario 1: Citrix XenMobile or CEM and switching from MDX technology to MAM SDK policies causes different behavior regarding the MAM SDK enabled applications and the web sso policy won’t work or connections won’t be active.

Notes from the lab: Citrix StoreFront 2203 and the cannot complete request

Sun, 01 May 2022 08:05:13 +0000

A quick blog regarding my Citrix lab upgrade from Citrix Virtual Apps and Dekstops (CVAD) 1912CU4 to 2203 and the little StoreFront snag I hit.

Summary of my setup: Two Delivery controllers Two StoreFront servers cohabitating with Director as well Two FAS servers Two WEM servers One unmanaged VDA worker

And a Citrix ADC HA setup load-balancing the entire solution with keep in mind only TLS1.2 enabled for the services.

Notes from the field: Citrix Files / ShareFile MDX SSO not working

Wed, 20 Apr 2022 17:29:45 +0000

At my latest Citrix Endpoint Management customer there were some issues regarding Citrix Files / ShareFile not achieving an SSO throughout the MDX/MAM enabled applications. Everything outside the MDX/MAM application bubble would work just fine only when tunnelling through the internal only application this would fail. The setup was comprising of a dual IDP setup with Microsoft ADFS and Citrix Endpoint Management itself.

First thing to note was the ACL regarding the customers CEM environment and allowed IP-addresses. Adding those would instantly resolve the messages seen in the debugging logs of IP-address not on the allow list.

Notes from the field: VMware Horizon instant clone breaks with Kerberos armoring

Sat, 19 Feb 2022 08:39:19 +0000

On my current customer project we’ve encountered a strange issue when some stricter security policies were implemented. Kerberos armoring was enabled which effectively broke the instant clone process for Windows 10 1809/1909 releases but not for 2009 or 21H2.

It all started with a ticket that the image update process in Horizon would error out and fail constantly on the mentioned images. On the newer builds no problem at all. At first we thought it was an Microsoft update of some sorts but after some troubleshooting with colleagues Wesley Kieffer and JP Ruitenbeek it turned out to be new hardening items which got turned on.

Notes from the lab: Citrix ShareFile and VMware Access SSO

Sun, 13 Feb 2022 11:36:39 +0000

When configuring Citrix ShareFile for an SSO experience with your Microsoft Active Directory setup we have the following guides to use it from Citrix. See How to Configure Single Sign-On (SSO) for ShareFile (citrix.com)

Well I’m having my setup with another Identity Provider in my own lab and still want to achieve an managed SSO setup from my end. To get this to work I checked the setup from an existing integration setup like Microsoft ADFS and reverse engineered it to VMware Access instead.

Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

Mon, 20 Dec 2021 11:04:41 +0000

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

Notes from the field: Another cannot complete your request with Citrix FAS

Sun, 14 Nov 2021 14:58:41 +0000

We’ve all seen it time and time again some misconfiguration with Citrix StoreFront and/or Citrix FAS and you’ll be getting the cannot complete your request message in your screen. Digging in the StoreFront logs and you’ll be seeing the most interesting messages of error kind in which you would think am I a rocket professor?

My story for this certain scenario would be a CVAD deployment integrated with FAS and everything working just fine with some minor bumps like adding your resources to the Windows Authorization Access Group and magic occurs things start to work. See Cannot Complete Your Request Error only occurs to certain users connecting from ADC with Azure MFA over to Storefront (citrix.com) for the fun of it and it’s buddy Common Resolutions to “Cannot Complete Your Request” Error when connecting directly to StoreFront Server (citrix.com)

Notes from the lab: VMware UAG content gateway and an A+ rating

Tue, 31 Aug 2021 20:23:07 +0000

In addition to Jesper Alberts his blog a follow up with another custom UAG edge service which has it quirks called the content gateway. For the SEG article see vJAL.nl - Secure Email Gateway

Now diving in, when you configure the edge service you have the following options to configure Custom Values for Content Gateway and bare in mind that you’ll find this article after your first check on SSL Labs because an disappointing rating is what you get out of the box. See below screenshots for an A+ rating on SSL Labs:

Notes from the field: VMware Access Kerberos integration and Office 365

Fri, 20 Aug 2021 20:15:41 +0000

Okay let’s say you have your setup for VMware Access nicely configured with your directory search attribute configured as userPrincipalName because that’s the modern way with all cloud services etc. and configured your inbound Kerberos authentication through the IDP of the Access connector. Everyone is happy and all is working well with external connections, internal connections, mobile connections and what other type of connections we can think of. Then comes the day Office 365 is going to be integrated and still all is working well externally, mobile as well and then you get some calls regarding users who get a prompt unknown user when accessing the portal through Kerberos logon. You get to the trusty old log view and dig in and see message unknown user entries with the UPN value of your internal domain. Well, turns out that when the search attribute is selected as UPN you cannot switch over to your routable domain which is being used in Office 365 and still expect a working Kerberos logon. The only way this little beauty is going to work if is the search attribute is sAMAccountName. After a GSS support case got this one confirmed this is the only way that will work, or you would need to add a global catalog specifically for the domain in question which means double accounts, dedicated domain controller etc. etc. no one wants that!

Notes from the field: VMware Workspace ONE UEM and Android Zero Touch

Sat, 14 Aug 2021 08:21:19 +0000

On a recent project we were implementing Android Zero Touch for out of the box enrollment through WS1 UEM. For a detailed explanation what Android Zero Touch is take a look at the following URL: Zero-touch enrollment for IT admins - Android Enterprise Help

When the Zero Touch Portal is enabled through the reseller and you have your access the DPC part of UEM or any other supported EMM vendor can be added and assigned. For WS1 UEM we have the following options for configuration: Enroll Android Device Using Zero Touch Portal

Notes from the field: VMware Access with VMware UAG and JWT validation

Sat, 07 Aug 2021 08:20:03 +0000

It’s been a while since I’ve retested the setup with validating gateway request with JWT entries, because I thought it was depending on an appliance such as F5 for it to work. See Launching Horizon Resources Through Validating Gateways (vmware.com)

I did try and configure it none the less but never got it farther then just enabling JWT in Access with no audience enabled and the UAG also not configured with any WS1 for a working desktop, otherwise it would always error out with something like below:

Notes from the field: VMware Access Roles and RBAC bug

Sat, 07 Aug 2021 07:44:37 +0000

On recent projects we where configuring RBAC roles in VMware Access Cloud and stumbled across something annoying which turned out to be a bug. The issue is that when you assign the RBAC roles through super admin, read only admin and directory admin that once added you can’t delete or re-add the same group, it will error out with the following error:

It also isn’t possible to unassing the role anymore, and you might think okay well the role still works! Well no it doesn’t the role is hardcoded and can’t be removed anymore: Deleting the complete directory and re-adding the directory doesn’t solve it either it will come back no matter what. Logged a GSS support case for this and it turns out this is indeed a bug.

Notes from the lab: VMware UAG 2106 and Admin SAML

Mon, 26 Jul 2021 20:44:36 +0000

VMware introduced SAML login capabilities for the admin facing side of UAG with version 2106. See the following article: Release Notes for VMware Unified Access Gateway 2106

This quick home lab blog shows how easy it is and how to integrate this with VMware Workspace ONE Access as your entry point.

First things first, before we start you should have the IDP.xml file of your IDP in place if this is a VMware Access setup or Microsoft ADFS it doesn’t matter, the flow is exactly the same. You upload this at the identity bridging settings part of the UAG.

Notes from the field: VMware UAG and Citrix ADC scenario's

Thu, 13 May 2021 21:33:15 +0000

On a recent project we were testing some scenario’s for the usage of VMware Blast BEAT through Citrix ADC. For some more information regarding Blast see the following article: VMware Blast Extreme Optimization Guide | VMware

Normally you would see that the Citrix ADC setup is an SSL-BRIDGE vserver with accompanying UDP vserver on the same IP for the ports 443 and 8443 which are the default for BLAST and BEAT usage.

Notes from the lab: VMware vCenter 7u2 ADFS changes

Sat, 20 Mar 2021 09:27:38 +0000

When vCenter 7 introduced ADFS integration I jumped on the configuration part in my lab and set it up with the necessary OAUTH integrations:

Now with vCenter 7u2 there are some changes when you have it in place and are upgrading:

The trust store is changed to VECS and you need to change/add that in vCenter:

Well one would think that everything is nice and dandy after this but I completely forgot that at the time I set the whole ADFS integration part on LDAP and of course no signing requirement in place:

Notes from the lab: Citrix ADC and VMware ESX 7u1/7u2

Sun, 14 Feb 2021 20:49:15 +0000

First things first. Citrix ADC at this time isn’t supporting VMware ESX 7.0.1 according to the following article: Support matrix and usage guidelines (citrix.com)

This is something that obviously will get supported in due time. But for the people who are running it just as I am in the lab you would see issues like the ADC instances would lose connectivity or will not load the appropriate network drivers at boot. This is because of the VMXNET3 interface which is causing issues.

Notes from the field: VMware Horizon sessions disconnecting after syslog changes on UAG

Sat, 13 Feb 2021 21:04:24 +0000

On a recent project where we have VMware Horizon 7.13 and UAG 20.09 appliances for the external connections some strange behavior was observed when putting in the syslog URL entries.

After adding or removing entries here and saving the settings all the connections through the UAG will get terminated. Finding this behavior strange as to you don’t do anything that special so why disconnect these sessions.

Had some discussion on the vExpert slack channel and quickly came to light that it looks like an regression issue.

Notes from the lab: Bye Bye VMware View Composer

Mon, 18 Jan 2021 22:00:26 +0000

I was upgrading my lab to VMware Horizon 2012 and yes shame on me I still had an composer in my setup. It was already mentioned that VMware Composer is deprecated from the 2006 release but now in 2012 it will block your upgrade when you still have it enabled. Only after disabling composer on your vcenter the upgrade will succeed and afterwards composer will be gone as an configuration item.

Notes from the field: Citrix StoreFront forcing connections through Citrix Gateway

Mon, 18 Jan 2021 21:53:49 +0000

On a recent customer project there was the need to migrate off of VDA TLS encryption and migrate the connections from StoreFront to Citrix Gateway.

The customer previously had StoreFront direct connections and used the VDA TLS encryption setup to provide a TLS encrypted session to the desktop or applications.

The VDA TLS encryption setup was too much engineering labor for the day 2 day operations and therefore they asked for a alternate solution but still provide the client>desktop as an TLS encrypted session.

Notes from the field: VMware Horizon Instant Clone and Imprivata OneSign

Thu, 07 Jan 2021 12:10:36 +0000

On a recent project consisting of an VMware Horizon instant clone setup and Imprivata OneSign in the desktop for SSO capabilities I’ve encountered some strange timing issues.

Normal logins through the horizon client via connection server would be ok with the OneSign agent online, logins through the UAG without TrueSSO would also be okay. (so it seemed)

TrueSSO enabled because the total solution is a Workspace ONE deployment and we want to use one login of course regarding credentials introduced a problem. The login process would work just fine but the agent of Imprivata would stay offline only to be online after a reconnect to the desktop.

Notes from the field: Citrix FAS request not supported

Sun, 11 Oct 2020 18:40:28 +0000

On a recent Citrix FAS deployment I’ve encountered the following error: “Request not supported” when logging in to a published application or desktop. Article https://support.citrix.com/article/CTX218941 explains that re-enrollment of the domain controller authentication template or another custom template for Kerberos usage should resolve the error.

A little bit of a background on the environment, an already working Microsoft ADCS environment was in play and in use for other services. From a design/security perspective it was designed that two dedicated Microsoft ADCS servers would be used and two Citrix FAS servers connecting these new servers. The setup was working as expected but only above error would keep coming when trying to access an application or desktop.

Notes from the presentations: Modern authentication glued together with Microsoft, Citrix and VMware

Mon, 05 Oct 2020 18:22:02 +0000

Very happy to share my first presentation on Virtual Expo with Erik Bakker, please click the following link for the recording and all other recordings as well.

https://xenapptraining.com/members/virtual-expo/2020-09/

Notes from the lab: Microsoft ADFS and VMware UAG

Sun, 04 Oct 2020 13:40:14 +0000

You don’t see many configuration articles around ADFS and UAG and that’s why I would like to share my setup.

First things first, I’m expecting that there is an working Horizon environment with True SSO enabled for access to the desktop. And a working ADFS environment to add a new application to test with.

My setup: 1 x ADFS for internal usage 1 x WAP for external usage 1 x UAG v3.10 – dedicated for ADFS with its own URL 1 x UAG v3.10 – dedicated for WS1 with its own URL 2 x Horizon Connection Servers 2 x Horizon Enrollment Servers

Notes from the field: Citrix FAS SSO not working with invalid CRL

Sun, 20 Sep 2020 19:02:52 +0000

Recently I got contacted by a customer who had problems performing an SSO to a newly build desktop environment.

The setup a greenfield resource domain and forest trust from an existing tenant with a two way trust. Basically everything was correct but the logon from the users would always get terminated at the desktop with invalid credentials.

After a short discussion and remote session and the error messages in the logs with an invalid CRL it was clear that was the issue. Troubleshooted the AIA/CRL locations and basically the defaults where still in play, explained that default push in AD isn’t a recommended approach. If any client can’t access the CRL it will give a deny on further actions (and other clients that don’t understand AD or are joined to AD won’t work as well).

Notes from the field: VMware UAG reverse proxy why doesn't it work!

Sun, 13 Sep 2020 09:06:52 +0000

When configuring VMware UAG as an reverse proxy I’ve encountered some issues last year that as far as I could see wasn’t all to well documented. My reference article for the configuration was the following: https://techzone.vmware.com/configuring-web-reverse-proxy-identity-bridging-vmware-unified-access-gateway-vmware-workspace-one-operational-tutorial#985671

Basically when you follow it to the letter in your test deployment and with a test site you will not have a working reverse proxy URL. At the time when I encountered this I’ve logged a GSS support case and in the troubleshooting process it was clear that the proxy pattern set wasn’t working whatsoever, the correct one should be (|/(.)|) instead of (|/intranet(.)|)

Notes from the field: VMware Horizon Enrollment Server and Core O/S

Thu, 20 Aug 2020 16:44:56 +0000

Recently had an deployment with a customer who has a mandate core o/s deployments are preferred unless the product doesn’t support a core o/s installation.

Well for this deployment we created two core o/s subordinate ADCS servers with the enrollment server software installed and configured. Everything is working fine and dandy, no issues and seems like its golden for production deployment.

Guess again.. all the Horizon products aren’t supported for a core o/s installation, yes it will work most of the time but if you’ll find any errors or in need of GSS support then most likely you would need to install a GUI variant. I’ve had two support cases open and both of them state it’s not supported even though the documentation isn’t explicitly stating that.

Notes from the field: VMware Access connector support LDAP Signing and Channel Binding

Thu, 20 Aug 2020 16:32:06 +0000

Quite recently I’ve encountered a random synchronization error that VMware Access connector could not synchronize and would error out with the following error: “Connector communication failed because of invalid data: The specified Bind DN and password could not be used to successfully authenticate against the directory”

At first I stumbled upon the known issues list: https://docs.vmware.com/en/VMware-Workspace-ONE-Access/19.03/rn/VMware-Identity-Manager-1903-Release-Notes.html#knownissues and checked if the computer name was the same as the name in the domain field and that was all correct.

Notes from the field: Citrix XenMobile / Endpoint Management Per App VPN not working for iOS

Mon, 17 Aug 2020 19:19:47 +0000

This was quite a nice one to troubleshoot, turns out there is a new configuration point for per app VPN and iOS devices, at least it was for me.

If you follow the configuration at https://www.citrix.com/blogs/2016/04/19/per-app-vpn-with-xenmobile-and-citrix-vpn/#:~:text=With%20the%20iOS%20per%20app,applications%20installed%20on%20the%20device. you’ll end up with a config that won’t open up a VPN when accessing the browser. Solution for this is to change the default provider type in the policy from App Proxy to Packet tunnel also mentioned here https://docs.citrix.com/en-us/xenmobile/server/policies/vpn-policy.html and explained it means the following:

Notes from the lab: Windows firewall profile not correct after reboot

Sun, 02 Aug 2020 06:54:05 +0000

Just thought of leaving a quick win here. Did you ever had the firewall profile of Windows not correctly mapped after reboots etc.?

This is because after a reboot the Domain Controllers put it in e.g. public profile and this will get passed on to other servers as well. This will effect in not being able to manage machines because of firewall blocks etc.

Solution is to restart the “Network Location Awareness” service and dependent “Network List Service”. This will reset it to domain profile and after reboots of the other machines which have this it will be updated to domain profile as well. Or restart the service as above that will also do the trick.

Notes from the lab: Citrix ADC IP Reputation

Fri, 03 Jul 2020 19:26:06 +0000

I’ve been playing around with the Citrix ADC IP Reputation feature - https://docs.citrix.com/en-us/citrix-adc/13/reputation/ip-reputation.html in the lab for some time and to be honest it’s such a small but very effective feature which I almost never see active, why is that?

If you’ve gotten a premium licensed ADC appliance it’s a simple right click>enable and you put in the necessary arguments in a responder policy. See the following article for a quick how to video - https://www.youtube.com/watch?v=WedxwiEVuG4 and basically that is it. The requests are going to be filtered on a Webroot service provider for malicious IP database and you can then drop those from ever getting at you network. (and put in a nifty log action so that you can filter as a syslog entry in Citrix ADM

Notes from the lab: VMware Horizon and Microsoft MFA NPS Extension

Sat, 20 Jun 2020 11:52:16 +0000

In my own lab environment I have a mixture of EUC components and dual factor configured accordingly, but more and more I see that customers also just use the MFA solution of Microsoft to integrate it for their environments. Why not it’s included with your license right.

So back to the techie part I’ve configured my own NPS setup on a Windows Server 2019 and configured the RADIUS setup. Installed the MFA NPS extension and had a pre-existing configuration for my Citrix ADC appliance.

Notes from the field: The unexplained Outlook pop-up

Sat, 20 Jun 2020 10:50:23 +0000

Quite recently I’ve had an interesting troubleshoot at a customer. The problem was at first that there was an issue in the newly build Exchange 2019 environment that Outlook clients would open up and ask for credentials in a domain joined environment, so the SSO part of WIA isn’t working and it “seemed” to work after you would put in credentials.

Long story short support case at Microsoft was in play and after some weeks of log troubleshooting no results and a standstill for the customers migration project.

Notes from the field: Configuring AFAS Online with Azure

Sat, 25 Apr 2020 18:25:55 +0000

I have a quick win for those who are also in the process of migrating an ADFS configured AFAS Online setup to Azure Active Directory. I’ve already had an support call with them and besides the point they don’t support any troubleshooting IDP setups they did their best which in turn got me to sharing this.

So down to the point, the following article describes the SSO needed part for AFAS Online: https://help.afas.nl/help/EN/SE/plv2_Config_SSO.htm

Notes from the lab: Configuring vCenter 7 with ADFS

Sat, 11 Apr 2020 11:54:52 +0000

With the release of vCenter 7 you can now integrate it with Microsof Active Directory Federation Services (ADFS)

See the following blog article for an overview: https://blogs.vmware.com/vsphere/2020/03/vsphere-7-identity-federation.html

See the following configuration articles for a setup overview: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-C5E998B2-1148-46DC-990E-A5DB71F93351.html https://kb.vmware.com/s/article/78029

With this information I’ve configured my lab environment to a working SAML based login with a few minor issues.

I had my ADFS setup load balanced through a content switching setup for external access. This is working great for my simple office 365 integration point but not so much if you’re trying to do more.

Notes from the lab: Migrating Windows vCenter to VCSA 7

Sat, 11 Apr 2020 11:22:08 +0000

In my lab environment I was running Windows vCenter 6.7 and with the release of vCenter 7 a migration is needed because there is no Windows vCenter anymore.

The following articles will give you enough information on how the process works especially the how-to from Vladan Seget: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-9A117817-B78D-4BBE-A957-982C734F7C5F.html https://www.starwindsoftware.com/blog/how-to-migrate-vmware-vcenter-from-windows-to-vcsa-6-7-update-1

Basically the process is the same for vCenter 7 with in my case one issue.

At first try my migration failed at the stage when the migration assistant is shutting down the Windows vCenter and the VCSA 7 is being brought up with the original IP, hence resulting in a conflict and the upgrade is in a broken state. I followed the process and saw that especially Windows Server 2016 has the annoyance to delay the shutdown for minutes, this is a known issue and happens from time to time and to my knowledge has to do when updates were installed (even after multiple reboots).

Notes from the lab: Citrix ADC Native Push OTP not working

Sat, 11 Apr 2020 08:37:15 +0000

I’ve updated my lab environment with Citrix Gateway push OTP support and had some trouble in configuring the Citrix SSO app on my iPhone. For some reason it couldn’t setup the gateway connection and it wasn’t reachable. (Well that was my bad in checking all my devices but I’ll get to that)

Before the push OTP change I’ve worked with the authenticator app behavior and put in the code myself and this worked fine. The change to push OTP isn’t too difficult, and the following articles give you plenty of how-to information: https://docs.citrix.com/en-us/citrix-gateway/13/push-notification-otp.html https://docs.citrix.com/en-us/citrix-adc/13/aaa-tm/native-otp-authentication/otp-encryption-tool.html https://www.carlstalhood.com/native-one-time-passwords-otp-citrix-gateway-13/

Notes from the field: Cannot access Citrix ADC or create HA set

Sun, 22 Mar 2020 23:58:28 +0000

Quite recently I was at a customer where they had an SDX setup with single instances and needed to be upgraded and converted to an HA setup.

Well easy does it I created the instances on the second SDX and started creating HA sets. Numerous went fine and then one started giving errors. Could not propagate from the primary and after checking SSH/SCP access this would fail as well. I logged in through the console of SDX/SVM and saw that the sshd daemon wasn’t starting anymore. (On a side note all of the original SDX instances were upgraded in regard to the exploit of last December)

Notes from the field: Configuring SentinelOne SSO with VMware Workspace ONE Access

Thu, 26 Dec 2019 19:07:44 +0000

SentinelOne’s configuration can be achieved after you have a valid account and support login. Afterwards its pretty easy to configure the SSO part.

In the cloud console of SentinelOne go to Settings»Integrations»SSO

Configure the following items for SSO usage:

IDP Redirect URL:

https://workspaceoneaccessurl:443/SAAS/API/1.0/GET/apps/launch/app/uniqueapplicationid

IssuerID:

https://workspaceoneaccessurl/SAAS/API/1.0/GET/metadata/idp.xml

Configure the rest of the items at your own requirements but don’t forget to upload the IDP public certificate of Workspace ONE Access.

Make copies of the Assertion Consumer Service URL and SP Entity ID to use in Workspace ONE Access.

Notes from the field: Configuring Autotask PSA with VMware Workspace ONE Access

Thu, 26 Dec 2019 18:44:50 +0000

Autotask PSA SSO configuration can be found at the following url: https://ww13.autotask.net/help/Content/AdminSetup/1FeaturesSettings/ResourcesUsers/Security/SSSO_OIDC.htm

For the configuration part of Workspace ONE Access SSO you can see the available API at this url: https://code.vmware.com/apis/57/idm#/

The problem is that Autotask PSA SSO doesn’t work/supports the setup of VMware Workspace ONE Access. I worked around this issue by having a federated setup to our Office 365 tenant and adding the Autotask application there and ultimately publishing the application as a custom application link and still provide the requested SSO.

Notes from the field: Configuring OpsGenie (without Atlassian Access) with VMware Workspace ONE Access

Thu, 26 Dec 2019 18:32:44 +0000

OpsGenie can use SAML SSO without the use of Atlassian Access, see the following url: https://docs.opsgenie.com/docs/single-sign-on-with-opsgenie

For the configuration part of Workspace ONE Access just add a new manual SAML 2.0 application and provide the following information according to above article:

- **Single Sign On URL** https://app.opsgenie.com/auth/saml?id=**”uniquesamlidprovided**
- **Recipient URL** https://app.opsgenie.com/auth/saml?id=**”uniquesamlidprovided**
- **Application ID** https://app.opsgenie.com/auth/saml?id=**”uniqesamlidprovided**
- **Username Format** = Unspecified

Username Value = ${user.email}

Notes from the field: Configuring Atlassian Access with Workspace ONE Access

Thu, 26 Dec 2019 18:27:23 +0000

Atlassian Access is the SSO portal being used for SSO access across Jira, Confluence etc. for the configuration part see the following url: https://confluence.atlassian.com/cloud/saml-single-sign-on-943953302.html

For the configuration part of Workspace ONE Access just add a new manual SAML 2.0 application and provide the following information according to above article:

- **Single Sign On URL** https://auth.atlassian.com/login/callback?connection=saml**”uniquesamlidprovided**
- **Recipient URL** https://auth.atlassian.com/login/callback?connection=saml**”uniquesamlidprovided**
- **Application ID** https://auth.atlassian.com/saml/**”uniqesamlidprovided**
- **Username Format** = Unspecified
- **Username Value** = ${user.email}
- **Relay State URL** = [https://id.atlassian.com/login](https://id.atlassian.com/login)

Add the custom attribute mappings for firstname, lastname and userprincipalname.

Notes from the field: vCloud usage meter doesn't meter NSX

Thu, 26 Dec 2019 17:49:24 +0000

A while back I had an support case with VMware support regarding NSX integration and that it wasn’t getting metered by vCloud Usage Meter in a customer deployment. Turns out that Usage meter looks for a Global Transport Zone before the discovery of a Universal Transport Zone and metering can occur. So if you are in a setup that only has Universal Global Transport Zones it is expected behavior to see no NSX monitoring hits being satisfied in Usage meter. This can be resolved by adding a Global Transport Zone as a fictive addition so that it will meter your setup.

Notes from the field: Windows 2019 Storage Replica lock-up on VMware

Tue, 26 Nov 2019 17:37:13 +0000

On one of my latest projects consisting of a new Windows Server 2019 setup on VMware and making use of Storage Replica in a server to server setup for replicating home drives and profiles I came across a random lock-up of the VM and by that inaccessible shares.

The setup was all working until the failover part. It seems there is an delay of some sort and the failover isn’t instant or takes a while to be active with the server being unresponsive and disconnecting any form of management to the VM in question(VM tools are not responding as well and console login will not work in this failover time). I’ve tried the actions again of doing a storage replica failover and I got an BSOD on the VM stating: HAL INITIALIZATION FAILED I’ve tried all of this in a separate test setup and had this working without any problems on Server 2016, and Server 2019. Only this time it gave me this strange behavior. The difference in my own setup is HW level 14 and this new one had HW level 15 and the hosts are 6.7 13981272 build and my own setup is 6.7 14320388 build (older builds have also worked fine for me)

Notes from the field: Hyper-V to VMware migrated VM's cannot install VMware Tools

Sat, 05 Oct 2019 12:35:55 +0000

One of my last projects I needed to convert Hyper-V VM’s to VMware, this all went fine with the offline capability of vcenter converter and the migration succeeded. Only after trying to install the VMware tools this would hang on starting the VGauth services and several other dependencies. For reference the VM’s in question are a mixture of 2008R2 / 2012R2. After some troubleshooting and searching the knowledgebase I stumbled across this article: https://kb.vmware.com/s/article/55798

Notes from the lab: Citrix ADC Native OTP and AdminSDHolder

Sun, 29 Sep 2019 18:35:08 +0000

While doing some lab work I came across an issue that the Domain Admin accounts could not register on the manageotp site while Domain Users could. This got me figuring it out.

For the use of Native OTP on the ADC we need to use an bind account for Active Directory which has the appropriate write permissions on the userParameters value of the users.

When we delegate control of the exact write permission of the userParameters everything is fine for normal users but administrator accounts won’t work. When we use a service account with full blown domain administrator permissions as the bind account then it works.

Notes from the field: vCenter cannot validate SSO domain

Tue, 30 Jul 2019 20:51:48 +0000

Came across a peculiar issue when adding an second vCenter to the same SSO domain and enable ELM. The first deployment worked like a charm and the second errored out with the following error: It turns out there is a known bug when using uppercase FQDN in the configuration wizard, the solution is to put it all in lowercase. see the following link for reference: https://kb.vmware.com/s/article/56812

Notes from the field: UEM/vIDM integration caveats

Mon, 24 Jun 2019 18:30:48 +0000

Not too long ago I encountered some issues when configuring UEM and IDM integration. When providing the vIDM URL in UEM for configuring the integration it would error out with below error:

After some troubleshooting it appeared that the access policies where not properly configured as in the last rule in the default access application ruleset was blocking access. Resolution was editing the default policy and ending it with the password method which is associated with the built-in workspace IDP, after that the integration part is working as expected.

Notes from the field: vIDM and o365 modern authentication delay

Sat, 08 Jun 2019 18:37:53 +0000

Just a quick win blog to mention and give a heads-up that when you are in the process of configuring vIDM and o365 you might encounter native clients prompting for authentication and a big ass delay when you flip over the authentication and the requested domain from managed to federated with vIDM. This might be up to eight hours!!! Thanks to the #community #vExpert that I got this answer quite fast because I recalled that Laurens van Duijn put something similar in the vExpert Slack group mentioning that he saw this kind of behavior.

Notes from the field: VMware vCenter /dev/mapper/core_vg-core full

Sat, 08 Jun 2019 17:58:45 +0000

Not too long ago I’ve encountered an vCenter instance blowing up the /dev/mapper/core_vg-core with gigabytes of java dump errors.. Just for reference the customers setup is an dual SDDC with respectively an vCenter at each site comprising of vCenter 6.5 U2 and embedded linked mode enabled.

In troubleshooting mode I’ve encountered the following two articles:

https://kb.vmware.com/s/article/2150731

https://kb.vmware.com/s/article/60161

Afterwards decided to open up a support case. This resulted in a session which stated that they had seen this sort of issues arising in 6.7u1 and higher which root caused against hardware level 13 for the appliance and WIA Active Directory integration.

Notes from the field: VMware vCloud Usage Meter vROps cleanup not working

Sat, 25 May 2019 08:56:02 +0000

If you ever are in the proces of cleaning up your vRealize Operations Manager instances and are using vCloud Usage Meter as well you might find yourself in a situation that Usage meter keeps referencing an old node which is deleted.

There is a nice explanatory blog available from VMware to resolve most part of this: https://blogs.vmware.com/vcloud/2018/01/updating-vrops-instance-vcloud-usage-meter.html

But if you find yourself in the situation that the old node is still there in Usage Meter but not referencing an vCenter this won’t help.

Notes from the field: vCenter VCSA 6.5u2 SEAT cleanup

Sun, 10 Feb 2019 18:02:56 +0000

This is a quick blog to show how an SEAT database failure can be cleared after an sporadic growth and increase to the events part of the SEAT DB in VCSA. I’ll explain the issue origin in an upcoming blog, but in a nutshell the 20gb was reached within six days and crashed the vCenter of a secondary site.

You SSH into the vCenter VCSA and enable shell and afterwards go to the vpostgres directory to complete the tasks. See below entries for reference and testing:

Notes from the field: Citrix ADC Gateway Native OTP with GSLB

Sat, 12 Jan 2019 22:42:09 +0000

Fun quick fact that I’ve encountered when deploying a ADC Gateway GSLB setup for a customer! You only have to enroll once with the nFactor/Native OTP on one of the ADC’s. (when having a Active Directory Domain across multiple datacenter sites)

The setup of choice:

Two ADC appliances in HA set on each site
GSLB enabled in active/passive mode for the Gateway across both sites
Native OTP enabled and active as the way for authentication
Active Directory Domain across two sites

There is no difference in configuration whatsoever because the magic of Native OTP depends on Active Directory.

Notes from the events: PubForum|E2EVC and VMworld 2018

Sat, 17 Nov 2018 08:41:55 +0000

PubForum|E2EVC - Athens

This was the place to be for the community and without marketing BS (except for the sponsor sessions of course but that also is more technical driven) Like always it starts the day before with networking with community peers, seeing old friends and making new.

The first day Alex opens with his EPIC introduction to PubForum|E2EVC like he always does, this time as a fallen god. Afther the introductions the first sponsor session of ControlUp started off which demos the product and all their features (this case the what’s new)

Notes from the field: vSphere 6 NVIDIA vGPU not working

Tue, 25 Sep 2018 18:53:28 +0000

Quite recently I’ve deployed a POC setup for a customer who wanted to leverage NVIDIA vGPU for their XenDesktop environment. In regards to all the prerequisites being met the VM’s wouldn’t boot when trying to test this on the base build of vSphere 6(the latest version that could be downloaded from the site) and the dedicated hardware.

After some time troubleshooting the issue was in the base build of VMware which was downloaded from the site. It included a hotfix which in turn would kill the vGPU support / integration. The resolution was updating the host to the latest level of patches. (the host at first was standalone being prepped to be inserted into the cluster of the customer, once joined update manager could do the rest)

Notes from the lab: vRealize Log Insight Cluster Upgrade 1-2-3

Sun, 17 Jun 2018 14:15:16 +0000

I must say I’m very impressed by the simpleness and stability at how VMware put the upgrade process in place for vRealize Log Insight.

First a little bit of a background of my deployment:

Three node vRealize Log Insight 4.6.0 cluster
Integrated Load Balancer (ILB) configured
vSphere 6.7 as hypervisor platform

I had the deployment running for a while and saw that 4.6.1 was available. Simple as that downloaded the upgrade .pak file from myVMware and logged in to my Log Insight cluster address, started the upgrade and got prompted to redirect to the master node for the upgrade progress, and simple as that nothing else to do! Either it works and every node will get rebooted automatically or it will fail and rollback all nodes.

Notes from the field: Ghost NIC on VMware

Fri, 15 Jun 2018 14:11:29 +0000

Quite recently I’ve encountered an issue/question at a customer which complained that two virtual machines had ghost NIC’s attached. Well it doesn’t always have to be hard in our line of work ?, after a quick look it was clear that there were snapshots in place for those VM’s with deleted old NIC’s attached.

Removal of the snapshot and the NIC’s were no more.

See the following reference screenshot of the ghost NIC and the distributed port group NIC:

Notes from the field: NetScaler VPX & Intel Xeon Gold

Wed, 23 May 2018 12:47:35 +0000

Quite recently I came across an issue when deploying a VPX instance on VMware 6.5, which resulted in a bug of the VPX image and underlying physical hardware. For reference the following hardware was backing the hypervisor: Supermicro SYS-2029U-E1CR25M Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz VMware ESXi, 6.5.0, 7967591 with vSAN NetScaler VPX 12.0 57.24nc

When deploying the VPX appliance it will get the default VM version 7 which needs to get upgraded to VM version 11/13 to support VMXNET3 NIC interfaces, well easily said and done configured the setup and booted the appliance and got stumped with the following error:

Notes from the field: NetScaler maxloginattempts

Mon, 09 Apr 2018 15:02:35 +0000

Came across a very peculiar issue at a customer in regards to the values:

- Max Login Attempts
- Failed Login Timeout

As soon as a value has been put in you could not reset it to the default value of 0, not from the GUI or CLI it would just not accept it as a value

After a support case and some days further the solution is maybe simple but I didn’t had it in mind, the simple unset command on the vpn vserver in regards to the maxloginattempts resolves it.

Notes from the field: uberAgent to the rescue!!

Fri, 16 Mar 2018 00:26:35 +0000

We all know it, the once in a while “it’s slow logging on..” and then it gets dropped at the escalation desk for a resolution. So I got the call for troubleshooting this issue. Since I knew from previous experiences that uberAgent is the troubleshooting tool you will want for this I contacted them and requested the consulting license at https://uberagent.com/ (thanks to Helge Klein) did the installation of Splunk / Uberagent and got myself a monitoring baseline to work with. A little background on the setup:

Notes from the lab: NetScaler VPX nsnet_connect prevents logon

Sun, 03 Dec 2017 16:25:55 +0000

When I started to rebuild my lab I came across the most strangest thing when configuring my NetScaler’s again. First a little background regarding my setup:

VMware ESXi 6.5u1 Hypervisors

NetScaler VPX 1000 Platinum Appliances

Distributed vSwitches with vlan trunks enabled

Dedicated NSVLAN for management (tagged)

Data transport vlan tagged

Whilst configuring and setting op the first and secondary nodes I’ve let the default appliance imports intact, that is 2vcpu and 2gb of ram and changed the E1000 nic’s to VMXNET3 and upgraded the VM compatibility format to the latest level. Nothing wrong here and started configuring both appliances with their NSIP’s respectively. Created the HA set and all was well.

Notes from the field: XenDesktop RemotePC and Multi Licensing

Fri, 17 Nov 2017 15:06:07 +0000

Recently I got involved at a customer location which was going to use Remote PC catalogs in combination with their XenDesktop / XenApp 7.15 environment. This was no problem whatsoever to configure but on closer testing I encountered a bug that when you create for example a delivery group called “Windows 10 Remote PC” and adding more than one desktop the second, third and so on would get the published name of the local computer name e.g. WSDELL34951 which doesn’t comply with a standard name. The following can be observed for the delivery group name:Normally you would see at “PublishedName” an empty value, to correct this take a note of the “Uid” number and put in the following command:In this case my id was 4, and voila this will correct the name in StoreFront like in the following screenshot:

Notes from the lab: NetScaler Secure Web Gateway

Mon, 11 Sep 2017 19:55:59 +0000

Finally had some time to implement the NetScaler Secure Web Gateway in my lab. I’ve put together a small document with my findings attached here: NFTL_TF_NSWG

comments or questions please let me know.

Cheers.

Notes from the lab: Exchange Server 2016 CU6 broken by default??

Sat, 09 Sep 2017 22:26:21 +0000

I came across the most peculiar issue I’ve seen so far with Exchange 2016. Installed a greenfield setup and the ECP/OWA page was broken by default with the following entry in event viewer:

Event code: 3005 Event message: An unhandled exception has occurred. Event time: 9-9-2017 22:26:57 Event time (UTC): 9-9-2017 20:26:57 Event ID: 53b3f1166cb147408cb97bc79483c3f5 Event sequence: 2 Event occurrence: 1 Event detail code: 0

Application information: Application domain: /LM/W3SVC/2/ROOT/owa-4-131494624100042355 Trust level: Full Application Virtual Path: /owa Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\owa
Machine name: EX01

Notes from the lab: Windows Server 2016 black screen when launching any application

Sat, 09 Sep 2017 19:45:24 +0000

I came across an issue in my lab environment where the screen will go black while launching a session on Windows Server 2016. This is with XenApp/XenDesktop 7.15 LTSR The following registry entry: DisableLogonUISuppression (D WORD Value 0) did not resolve the issue as stated in the following articles: https://support.microsoft.com/en-us/help/4034661/windows-10-update-kb4034661 and https://support.citrix.com/article/CTX225819

Ultimatly after some trial and error the deletion of all subkeys from below registry entries resolved it:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Connectivity HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration

Notes from the lab: Windows Server 2016 and MDT gen2 secure boot

Thu, 31 Aug 2017 22:22:00 +0000

For some upcoming projects and also lab use cases I’ve decided to brush up on some MDT/Automation tasks.

For this I’ve deployed a new Windows Server 2016 Hyper-V VirtualMachine Gen2 with secure boot enabled and installed MDT server and configured it to an up and running environment(yeah.. right).

At first I thought there were some inconsistencies with the installer because I kept getting an error on the windows overlay filter driver and it’s signature, didn’t pay much attention and kept going configured the MDT Deployment share and everything with it. Clicked on the update share item and…. boom kept getting an error on unable to mount the wim file of winpe and well a broken MDT setup..

Notes from the field: XenMobile Certificate Based Authentication lessons learned

Wed, 23 Aug 2017 19:37:55 +0000

Throughout the XenMobile deployments with Certificate Based Authentication(CBA) I came across some items which I thought was worth mentioning.

CBA up until Secure Mail 10.6.20 / Secure Hub 10.6.20 was requesting new certificates on SSL exceptions, in effect the exceptions were triggered on every SSL connection error that occurred and thus requesting a new certificate from the PKI, this got resolved in version 10.6.20 by not using Java codes anymore but instead reading the NetScaler Gateway error code which gets presented to the client.

Notes from the field: Be Proactive! Apple ATS is coming

Sun, 12 Mar 2017 16:09:43 +0000

For those who are not aware Apple has an upcoming change regarding App Transport Security (ATS) https://developer.apple.com/news/?id=12212016b The date it should be in effect was originally January 2017… but was pushed back for migration purposes, and the new date is yet a mystery.

It will have impact! Be proactive and check your XenMobile / NetScaler environments:

NetScaler 11.1 will be the preferred build for TLS1.2 and the ECDHE cipher suites
XenMobile 10.4 RP4 and XenMobile 10.5 have the TLS1.2 and ECDHE cipher suites (plus ATS hotfix)

Once ATS is enforced, Apple will require at least one cipher suite enabled from a specific list of cipher suites. Apple supported ATS cipher suites are: · TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 · TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 · TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 · TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA · TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 · TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA · TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 · TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 · TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 · TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 · TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

Notes from the field: NetScaler SDX LACP Flapping issue

Mon, 06 Mar 2017 13:10:59 +0000

I came across a peculiar issue regarding a new NetScaler SDX 14020 setup in combination with a Cisco Nexus C9372-PX-E and C9336PQ infrastructure, a new buildup of the SDX/VPX with multiple HA instances spinning and a working environment. LA sets configured for HA probes and everything nice and easy separated through vlan access. Long story short, at first it looked like a bug regarding the combination of NetScaler and Cisco: https://support.citrix.com/article/CTX215720 and created an support case with the follow ups with it, afterwards it seemed that the untagged management vlan setup was overlapping from data channels and the root cause for this was at the Cisco ACI side of things, the EPG(EndpointGroup) and BridgeDomain were overlapping in that case. The solution was to create a new and dedicated EPG/BridgeDomain for the data channels of the NetScaler.

Notes from the field: XenMobile Location services and SQL deadlocks

Fri, 10 Feb 2017 21:08:01 +0000

Came across a pretty specific issue in a large mobility environment regarding an old value from XenMobile 9 and still present in XenMobile 10, this is called device triangulation, with this the mobile service provider can triangulate the exact location from the device with constant updates regarding there location (this was an old value which was used with SMG and not applicable anymore). This can cause significant impact on your database server with deadlocks.

Notes from the field: Provisioning Services 7.11/7.12 TLS 1.2 issue

Fri, 10 Feb 2017 13:25:41 +0000

Citrix Provisioning Servers can be showing a offline status because the SQL native client version (11.0.2100.60) installed with it will not support TLS 1.2 and due to this it will give an error in event viewer with event ID 11 - Undefined database error

Installing the latest version of SQL native client on the PVS servers should resolve the issue.

Notes from the field: Quick win: XenMobile remove bulk redeemed enrollments

Fri, 10 Feb 2017 13:15:58 +0000

When you are using enrollment invitations and you don’t clean this up for let’s say an environment with a few thousand of users/devices this could be a time absorbing action to do. Luckily there is a quick win for this and you’ll want to create a query for “dbo.ENROLLMENT_PASS” on the Database server and remove those entries afterwards the redeemed invitations are gone.

Notes from the field: XenServer 7 mouse alignment MCS/PVS machines and XenServer 7 MCS XenTools

Wed, 01 Feb 2017 10:40:52 +0000

Came across two bugs on a XenServer 7 deployment in combination with XenDesktop/XenApp 7.12 worth sharing:

The first is a mouse alignment issue which results in VNC mouse pointer slowness or disalignment of the pointer on a console session in XenServer, the following can check the status of the usb and usb_tablet parameters on the vm’s:

xe vm-list uuid=[of the provisioned machine] params=platform

which will give the output of the VM and the following command will set the value’s:

Notes from the field: XenMobile CBA didn't I revoked that cert?

Sun, 30 Oct 2016 15:19:02 +0000

Just to start it off I’m assuming that the following is in place and fully configured and you are familiar with these concepts:

XenMobile 10.x cluster (XMS)
Active Directory (AD)
Active Directory Certificate Services (ADCS)
Active Directory Certificate Template(s)
NetScaler Gateway (NSGW)
Certificate Based Authentication (CBA)

Which all of them are combined in a XenMobile deployment which is configured to use CBA as an enrollment requirement.

I came across a limitation/by design issue in conjunction with the web enrollment of ADCS that XMS cannot solve, meaning that enrollment and requests for the first time will work just fine but when you revoke or selective wipe a device/user and the latter enrolls again you will get a cached certificate from XMS (you say what…) Revocation in XMS will work just fine but not at this point because according to support the API used in ADCS is not capable of doing a revocation, and basically XMS is using the web-enrollment for this and relying on that.

Notes from the field: XenMobile caveats

Sun, 10 Jul 2016 09:43:41 +0000

I’ve done a couple of Xenmobile implementations and found at least two interesting caveats that stood out, when implementing XenMobile and finding resolutions for the problems you’ll get when not adding it in your deployment.

No.1 NTP got introduced again with XenMobile 10.3.x to be configured in the appliance, a little tip enter in an reachable internal server, when you don’t pay attention and let it stay not configured for example on VMware you will get a very nice error message from time to time on the console of your VM: “hrtimer: interrupt took XXXXXX ns” (the xxxxxx is variable) this leaves your node in an failed state and the only resolution then is a reboot of the node.

Notes from the field: Netscaler Insight Centre not showing data

Tue, 05 Jul 2016 11:32:15 +0000

I’ve come across an issue regarding the Netscaler Insight Centre were data is not showing all the time, at random it just fails on reporting and shows nothing. It seems that after a talk with support there is memory corruption occuring when the usage of insights memory is above 75%.

Resolution shall be active in the 11.0.67.x release of the product.

Notes from the field: Netscaler Insight Centre

Tue, 07 Jun 2016 17:51:27 +0000

I came across an issue with Netscaler Insight with the latest build for Netscaler 11 and the same for Insight, logging did not reach the appliance regarding GUI flowcharts, we did see traffic generate from and to the Insight centre but no updates in the GUI screen, after some digging around and reporting this with Citrix it’s an bug regarding the Integrated Caching feature, this needs to be disabled otherwise it won’t work at all! ok.. that’s nice.. permanent fix is yet to be developed.

Notes from the field: XenMobile the road so far

Sat, 04 Jun 2016 11:28:55 +0000

Well it was time for an update regarding some XenMobile actions from the field, attached is an PDF with some of my ranting, enjoy the read:

Notes from the field XenMobile the road so far

Netscaler VPX Multiple PE

Tue, 03 Nov 2015 21:19:21 +0000

Came across this article today http://support.citrix.com/article/CTX139485 and basically changed my current vpx3000 2vpcu and 4gb to 6gb and 4vcpu and got the goodness!

XenMobile some field stuff

Fri, 23 Oct 2015 12:44:19 +0000

Thought I would share some XMS 10.1 knowledge:

the tool you need when creating apns: https://xenmobiletools.citrix.com/XenMobileCloudTools-3.0/home/

the tool you need when diagnosing the environment (still BETA): https://xmdiag.cgm.citrix.com/users/signin

the rolling patch #1 for XMS10.1 (cannot find this from the Citrix site but old trusty google does): http://support.citrix.com/article/CTX201757

If you use SSL offloading with the netscaler test, test and test the internal lb vserver for MAM if the 8443 will get passed through allright otherwise a broken MAM.

Citrix TAAS / CIS

Fri, 16 Oct 2015 12:07:20 +0000

A greatly unknown toolkit by many is the online Tools as a Service from Citrix where you can upload the dumpfiles of:

XenDesktop XenServer XenApp Netscaler Provisioning XenMobile ByteMobile CloudBridge CPBM CloudPlatform And there will run an automatic analysis of the uploaded dump files with common issues and best practices, a nice quick and easy debugging solution.

Take a testdrive @ https://taas.citrix.com or https://cis.citrix.com with your mycitrix account.