During my last implementation of UEM which was 1907 I encountered multiple error with tunnel , not the servers or configuration , its in console when I try to open the page it shows the error Workspace one  "Page not loading , Go back to the previous page or refresh the page to try again "  once I solved this issue I got another call from other colleagues for error "an error has occurred . something unexpected happened, if the issues persists , please contact you IT administrator

The Problem

Going to instigate these issues like any administrator start normally by checking the console logs and then API logs  ,technically you will not find much in console weblog but you may find exceptions and error in API logs , after day of investigation we found its happened because of communication issues  and definitely it's a communication issue with API which could happen when Console try to reach the API server and for some reason the traffic is blocked or routed to wrong directions , thinking for this way make me check the below :

  1. Check if there is a proxy configured in Concore or API servers (in case of API is not installed with console in same servers)
  • Check the Browses IE Proxy and remove the proxy if its configured
  • Check the windows service proxy : proxy can be configured in windows as services by command line or by group policy or third part tools , from command line use the command :

netsh winhttps show proxy

to remove proxy :

netsh winhttp reset proxy

  • Check if windows firewall is configure in servers : sometime windows firewall

blocking the traffic m so what do you need it to disable the firewall totally or add roles to firewall to allow the traffic flow

  • If the API server or console servers are behind load balancer : in this cases you need to verify:
    • Is there any security inspection in the route between API-to-console virtual load balanced URL or from console-to-API virtual load balanced URL : in sch cases call network and security team to solve this issue
    • How do you deal with certificate  traffic flow between servers which normally it would be :
mode Description Traffic flow Note
1 SSL pass through SSL terminated in the servers and load balancer will deal with it Traffic will reach and load balancer as 443 (no decryption or encryption will happen) then from load balancer to servers also as 443 with same certificate (server will decrypt the certificate ) In such case you need to bind certificate with IIS
2 SSL offloading SSL terminated in load balancer Traffic will reach load balancer as 443 , load balancer will check certificate decrypt the traffic , and then by default the traffic will be in port 80 from load balancer to servers This will be applied once you "API SSL Offloading " during the application server setup
3 SSL bridging SSL terminated in load balancer and servers will have certificate too (self-sign or public or from local CA) Traffic will reach load balancer as 443 , load balancer will check certificate decrypt the traffic  then re-encrypt , and then by default the traffic will be in port 443 from load balancer to servers In such case you need to bind certificate with IIS

Understanding the above will help you to know how to trace the traffic flow between servers  :

  • Within the API server try to reach the API server URL (http(s)://local host/API/help) it should show the API screen (http or https based on the SSL termination mode , mean in SSL off-loading it will be http and with other modes it will be https ), this just to verify API is working fine locally in each server
  • try to reach the API server load balanced URL (https://URL/API/help) it should show the API screen , if no, this mean you have an issue with load balancer or firewall (e.g.  load balancer IP in vlan and Servers in different vlan )and need network team help to solve it
  • in any of the above cases if API page show "No API found" this mean there is application firewall or URL filtering enabled in firewall of load balancer which need to be checked by network team or security team
  • IIS not fully installed : I noticed that some of engineers when they install API /Console services they don't verified the IIS in the application servers. And they depend on the installed to so , as the installer will not install IIS as per the requirements and always there will me missing features that need to be added ; this can be discovered  with Workspace one verification tools which is need to be done before the installation
  • After all the above , if still not working the suggestion to open call with VMware support to help and advice.

The Solution

you need to solve the above issues and once its settled its work by diagnostic or by working with support

to configure KEMP load balancer for VMware Workspace One Access (old Name VMware Identity manager vIDM)):

  1. You need the below certificates :
    1. KEMP Load balancer Root Certificate (if you configured Kemp to use your Public certificate as Wild card it will be your root certificate of the public certificate )
    2. All the Intermediate Certificate
    3. VMware Workspace One Access root certificate (you can get it from appliance Configuration, Install SSL Certificate )
    4. Be sure to have it in PEM format

in my case im using vIDM self-sign certificate and in vIDM ,be sure to import the KEMP load balancer root certificate under trusted CA, however you can use certificate  generated from your local CA but  you need to generate CSR file from vIDM then generate certificate and install it with the full chain in vIDM prior these step )

  • Import all of the above certificates :
    • under "Certificate and Security"   Intermediate Certificates
    • import all of the above certificates
  • Configure Virtual Servers

The following are the steps involved and the recommended settings to configure the VMware Identity Manager Virtual

1. In the main menu of the LoadMaster Web User Interface (WUI), go to Virtual Services > Add New.

2. Type a valid Virtual Address.

3. Type 443 as the Port.

4. Enter a recognizable Service Name, such as VMware Identity Mgr.

5. Click Add this Virtual Service.

6. Configure the settings as recommended in the following table:

Section Option Value Comments
SSL Properties SSL Acceleration Enabled
Reencrypt Enabled
Supported Protocols TLS1.0; TLS1.1; TLS1.2; TLS1.3 While this workload may not support TLS1.3 yet, Kemp recommend enabling it for future proofing.
Cipher Set Best Practices
Standard Options Persistence Mode Active Cookie
Timeout 1 Hour
Cookie name JSESSIONID
Scheduling Method least connection
Advanced Properties Add a Port 80 redirector VS https://%h%s Click the Add HTTP Redirector button. This automatically creates a redirect on port 80.
Real Servers URL /
HTTP Method GET
Real Server Check Method ICMP

7. Add the Real Servers:

a) Expand the Real Servers section.

b) Click Add New.

c) Type the address of the relevant Real Server.

d) Type 443 as the Port.

e) Complete the other fields as required.

f) Click Add this Real Server then click OK to the pop-up message.

g) Repeat the steps above to add more Real Servers as needed, based on your environment.

Create a HTTPS – Connection Server HTTPS HTTP Redirect Virtual Service

Clicking the Add HTTP Redirector button automatically creates a port 80 redirect Virtual Service. This is optional, but the purpose of this Virtual Service is to redirect any clients who have connected using HTTP to the HTTPS Virtual Service. Kemp also recommends changing the Persistence Mode and Real Server Check Method to None.

AirWatch implementation procedure  for less than 5000 users, considering the design with below configuration

  1. The implementation On-Premises NOT SaaS
  2. Database installed in Separate Server
  3. Console role hosted in separate Servers (local or DMZ) :
  4. Device Service /AWCM /API hosted in same servers

The procedure as below:

  1. Settings before setup
  2. Database : first step is to install the database
  3. Console Role
  4. AWCM Role\DS\API Server
  1. Prerequisites
  2. That need to be prepared in the server and need to be ready in advance include firewall roles , certificates ,load balancers and SQl server (mandatory)
  3. In addition all server need :
    1. Windows server x64 2008-2016 (2019 is not supported yet )
    2. to have .Net framework 4.62 and above
    3. to have .Net core
    4. all other staff the installation media can install it include IIS (IIS configuration manually is a must)
  4. Console setup :
    1. from the media start the installation
    2. Fill the database and URLs
    3. Select from roles Console only (remove API &DS )
    4. When the installation done go to IIS and bind the certificate (in case of SSL off-loading you may not do this step as certificate will be in load balancer)
    5. In case of multiple, Console servers , go to services and disable the below services in the (Second node)
      1. AirWatch Device Scheduler
      1. AirWatch GEM Inventory Service
      1. AirWatch Directory Sync
      1. AirWatch Content Delivery Service
    6. Configure the proxy (outbound connection incase it used for outbound connection)—in Global
    7. Create OGs and be sure the include OG-ID  for each OG(However its also preferred to keep it to last to avoid confession when configuring components )
  5. AWCM Role\DS\API Server:
    1. from the media start the installation
    2. Fill the database and URLs
    3. Select from roles (AWCM& API &DS )
    4. (For API)When the installation done go to IIS and bind the certificate (in case of SSL off-loading you may not do this step as certificate will be in load balancer) note that it will be port 80 by default in case of load balancer with SSl-offloading and restart the server after that
    5. Go to console and do the below:
      1. From Sites URl verify the  URl of API/DS/AWCM….all of them are same and should be
      1. Enable AWCM under site url …be sure that local port is 2001 and external port is 443
      1. From Groups & Settings > All Settings > System > Advanced > Secure Channel Certificate, Install secure channel certificate in order to activate the token
      1. Enable AWCM to Communicate with Devices,
        1. Navigate to Groups & Settings > All Settings > Device & Users > Android > Intelligent Hub Settings and scroll down to the AirWatch Cloud Messaging section.
        1. Select the Use AWCM Instead of C2DM/GCM as Push Notification Service check box to enable AWCM in the profile.
        1. The AWCM Client Deployment Type drop-down menu is automatically changed to Always Running and can no longer be modified.
      1. Generate Device Certificate
      1. Configure APN certificate
      1. Configure the Android EMM account
    6. After this we can go to configure connector then AD integration
    7. Restart the server

Verification phase

  • Open AirWatch Console
  • Choose About Airwatch – check the version
  • Check the Site Links: open Groups & Settings > All Settings > System > Advanced > Site URLs and look through the links
    • Only Peripheral Service URL should be "localhost"
    • Google Play has a defined port
    • Change SOAP and REST API URL links, instead of AirWatch Console URL put AirWatch Devices Services сервер – for example, instead of https://acme-console.com/AirWatchServices put https://acme-ds.com/AirWatchServices and instead of https://acme-console.com/API put https://acme-ds.com/API.
  • Check the connection with Device Services server with a defined in the install phase external URL, signed with external certificate (type of link: https://<DS_URL>/DeviceManagement/Enrollment  )
  • Check the AWCM component, using link https://<DS_URL>:2001/awcm/status*
  • Check AirWatch services – launch services.msc in Windows Server and check that AirWatch services are Started
  • Check the GEM Inventory Service: go to the AirWatch Console server, in the folder C:\AirWatch\Logs\Services\ and delete the file AirWatchGemAgent.log; open services.msc and restart GEM Inventory Service. New log will either NOT show up, or show up without errors.

you can go deep in verification using the below URLS

Device Services

Description URL Endpoint Status code
Device Services Enrollment /DeviceManagement/enrollment HTTP 200
App Catalog /DeviceManagement/appcatalog?uid=0 HTTP 200
Device Services WinMo Tracker /DeviceServices/tracker.aspx?id=0 HTTP 302

Console

Description URL Endpoint Status code
Web Console v9.2+ /AirWatch/login HTTP 200
Web Console (pre-9.1) /AirWatch/login HTTP 401

API

Description URL Endpoint Status code
API /api/help/#!/apis HTTP 200

AWCM

Description URL Endpoint Status code
AWCM /AWCM/Status HTTP 200

Secure Email Gateway

Description URL Endpoint Status code
ActiveSync Connectivity /Microsoft-Server-Activesync HTTP/1.1 401

VMware Tunnel – Proxy Component or Unified Access Gateway (Tunnel)

Description URL Endpoint Status code
HTTPS https://<TUNNEL_URL>:<HTTPS_Port> HTTP 407

Content Gateway or Unified Access Gateway (Content Gateway)

Description URL Endpoint Status code
Content https://<Content_Gateway_URL>/content/systeminfo HTTP 403

ENS V2

Description URL Endpoint Status code
ENS V2 /MailNotificationService/api/ens/alive HTTP 200

Error Message

1920 1.png


Error 1920. Service AirWatch Batch Processing service(AirWatch "Service Name" service) failed to start. Verify that you have sufficient privileges to start system services.

Cause
This message come up in air-watch installation when the application server is trying to start the Airwatch related services and does not have connectivity to signing.awmdm.com and timeout value passed and fail to start the service

Solution (1)
Continue clicking on ignore button to complete the installation however it will take time and you need to keep pressing the ignore button

Solution (2)

You might manually need to invoke the CertificateInstaller.exe residing in Workspace ONE UEM/Supplemental Software/CertInstaller/.

Follow the instruction provided in the section Generate Installation Token from myAirWatch: Manual Method from the document VMware Workspace ONE UEM Installation Guide or refer to the instructions below…

Generate Installation Token from myAirWatch
Toward the end of your Workspace ONE UEM installation, you may see a Global Enterprise Manager screen asking for your Installation Token generated from myAirWatch. This token is used to provision the necessary secure channel certificate to your Workspace ONE UEM database if it is not already present, such as in a new installation.
If your Workspace ONE UEM application server does not have outbound Internet access to the signing service, as defined under Network Requirements, then the Authentication Token field does not display on the Global Enterprise Manager. In this case, the manual flow installer is automatically launched. In case the installer is not automatically launched, you can manually run it by navigating to Workspace ONE UEM/Supplemental Software/CertInstaller/ and running CertificateInstaller.exe.

This EXE file opens a screen to guide you through the manual installation method.
1. Select Next to continue and start the wizard.
2. Select whether to use SQL Authentication or Windows Authentication. Select the same option that you chose during the main installation procedure. For SQL Authentication, the appropriate credentials are seeded in your config file. For Windows Authentication, you must enter the credentials of the Windows user to authenticate.
3. Select the Get File button and generate a PLIST file that contains a batch of certificate signing requests. Save this file to a location that has outbound Internet access to the myAirWatch signing service.
4. Log in to myAirWatch and navigate to Hamburger menu > myAirWatch > My Company.
5. Select Certificate Signing Portal.
6. Select Authorize Install.
7. Select Upload Your File.
8. Using the link, upload a PLIST file from your computer and select the PLIST file you saved previously.
9. Select Upload This File and save the file provided.
10. In the installer, select Set File and select the file myAirWatch provided. If successful, the success screen displays
11. Click Finish
12. Verify if the services had started.

Solution (3)

prior the installation of the application server ,Change the Windows service timeout value in the registry. You can use the following CMD command to update it for you. Make sure to right click and run CMD.exe as administrator. You can reference the following Microsoft KB for more information on this change: https://support.microsoft.com/en-us/kb/922918

"REG ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control /v ServicesPipeTimeout /t REG_DWORD /d 120000"

and you finish the setup you must go though tificateInstaller.exe process mentioned in solution 2 above