Published on November 7, 2019 - 5:33am
Last updated November 28, 2019 - 1:17pm

Login to like
Question

Pega 8.3.1 Upgrade is failing

Hi,

We are upgrading from Pega 7.2.2 to 8.3.1 via in-place manner and upgrade script gets failed after running few minutes. Below is the error coming in the log:--

Using Java 5 Management beans for process CPU timings: 2655496200000

[java] 2019-11-06 16:37:51,621 [ main] [ ] [ ] [ ] (.access.PageDatabaseMapperImpl) INFO - DeflateStreams is turned ON.

[java] 2019-11-06 16:37:55,285 [ main] [ ] [ ] [ ] (pboard.StorageStreamCommonImpl) INFO - engineCodeVersion: 08-03-02

[java] [SHUTDOWN]-----LogManager.shutdown()

[java] hasShutDowntrue

[java] Nov 06, 2019 4:39:47 PM com.pega.pegarules.internal.bootstrap.PRBootstrap

[java] SEVERE: Problem during method invocation (main)

[java] java.lang.NullPointerException

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.unregisterDbName(ConfigurationMapImpl.java:322)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.registerDbName(ConfigurationMapImpl.java:199)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.registerDbName(ConfigurationMapImpl.java:148)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.initializeSetOfDbNames(ConfigurationMapImpl.java:136)

[java] at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.initializeConfigurationMaps(ConnectionBrokerAdapter.java:202)

[java] at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.initializeConfigurationMaps(ConnectionBrokerAdapter.java:193)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.getConnectionBroker(PRDataProviderImpl.java:757)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.getConnectionManager(PRDataProviderImpl.java:724)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.isMultiTenantEnvironment(PRDataProviderImpl.java:1395)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.setStageDirectory(ParUtilsImpl.java:225)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.copyFileToVfsRoot(ParUtilsImpl.java:338)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.copyFileToVfsRoot(ParUtilsImpl.java:331)

[java] at com.pega.pegarules.deploy.external.util.UpgradeUtil.main(UpgradeUtil.java:57)

[java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

[java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)

[java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

[java] at java.lang.reflect.Method.invoke(Method.java:508)

[java] at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:380)

[java] at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:422)

[java] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224)

[java] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273)

[java] at com.pega.pegarules.pub.PegaRULES.main(PegaRULES.java:85)

[java]

[java] Nov 06, 2019 4:39:47 PM com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega

[java] SEVERE: Problem during method invocation (main)

[java] java.lang.NullPointerException

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.unregisterDbName(ConfigurationMapImpl.java:322)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.registerDbName(ConfigurationMapImpl.java:199)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.registerDbName(ConfigurationMapImpl.java:148)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.initializeSetOfDbNames(ConfigurationMapImpl.java:136)

[java] at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.initializeConfigurationMaps(ConnectionBrokerAdapter.java:202)

[java] at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.initializeConfigurationMaps(ConnectionBrokerAdapter.java:193)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.getConnectionBroker(PRDataProviderImpl.java:757)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.getConnectionManager(PRDataProviderImpl.java:724)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.isMultiTenantEnvironment(PRDataProviderImpl.java:1395)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.setStageDirectory(ParUtilsImpl.java:225)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.copyFileToVfsRoot(ParUtilsImpl.java:338)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.copyFileToVfsRoot(ParUtilsImpl.java:331)

[java] at com.pega.pegarules.deploy.external.util.UpgradeUtil.main(UpgradeUtil.java:57)

[java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

[java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)

[java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

[java] at java.lang.reflect.Method.invoke(Method.java:508)

[java] at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:380)

[java] at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:422)

[java] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224)

[java] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273)

[java] at com.pega.pegarules.pub.PegaRULES.main(PegaRULES.java:85)

[java]

[java] Exception in thread "main" java.lang.RuntimeException: Problem during method invocation (main)

[java] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:279)

[java] at com.pega.pegarules.pub.PegaRULES.main(PegaRULES.java:85)

[java] Caused by: java.lang.NullPointerException

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.unregisterDbName(ConfigurationMapImpl.java:322)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.registerDbName(ConfigurationMapImpl.java:199)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.registerDbName(ConfigurationMapImpl.java:148)

[java] at com.pega.platform.logicaldatastore.connection.internal.config.ConfigurationMapImpl.initializeSetOfDbNames(ConfigurationMapImpl.java:136)

[java] at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.initializeConfigurationMaps(ConnectionBrokerAdapter.java:202)

[java] at com.pega.pegarules.data.internal.store.ConnectionBrokerAdapter.initializeConfigurationMaps(ConnectionBrokerAdapter.java:193)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.getConnectionBroker(PRDataProviderImpl.java:757)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.getConnectionManager(PRDataProviderImpl.java:724)

[java] at com.pega.pegarules.data.internal.PRDataProviderImpl.isMultiTenantEnvironment(PRDataProviderImpl.java:1395)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.setStageDirectory(ParUtilsImpl.java:225)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.copyFileToVfsRoot(ParUtilsImpl.java:338)

[java] at com.pega.pegarules.deploy.internal.archive.ParUtilsImpl.copyFileToVfsRoot(ParUtilsImpl.java:331)

[java] at com.pega.pegarules.deploy.external.util.UpgradeUtil.main(UpgradeUtil.java:57)

[java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

[java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)

[java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

[java] at java.lang.reflect.Method.invoke(Method.java:508)

[java] at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:380)

[java] at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:422)

[java] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224)

[java] at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273)

[java] ... 1 more

***Moderator Edit-Vidyaranjan: Updated Platform Capability***

Installation and deployment
Upgrades
Correct Answer
AnandP
November 28, 2019 - 1:17pm

Hi,

This was a data issue. In our database, a corrupt/invalid instance of Data-Admin-DB-Name was found with all records as null, however the same was not visible within Pega. Since during upgrade pega validates all DB instances, this instance was causing the NullPointerException.

After deleting the record from database, upgrade started working.

View correct answer in context

Comments

Keep up to date on this post and subscribe to comments

MarcLasserre_GCS
Pega
November 7, 2019 - 5:51am

Hello, 

Do you mind sharing the full log.

  • reply

AnandP
November 7, 2019 - 6:57am
Response to MarcLasserre_GCS

Please find attached the full upgrade log

  • reply

KevinZheng_GCS
Pega
November 19, 2019 - 6:55am
Response to AnandP

Are you using your own prconfig.xml? I noticed these messages in the log:

[java] prconfig.xml missing key for initialization/settingsource
     [java] This is likely due to using a copy of prconfig.xml from a previous install.
     [java] >> assuming "initialization/settingsource=file"; Note that
     [java] this excludes use of entries in Data-Admin-System-Settings as prconfig settings

This can explain the NPE.

  • reply

AnandP
November 15, 2019 - 5:10am

Hi, Any update on the above?

  • reply

Anthony_Gourtay
November 15, 2019 - 7:20am

Have you tried resuming the upgrade?

Could it be that you lost your DB connectivity? or to your ldap?

Regards

Anthony

 

  • reply

AnandP
November 15, 2019 - 8:05am
Response to Anthony_Gourtay

yes, i have tried resuming the upgrade multiple times and it's giving the same error everytime at same stage. DB connection is successful everytime so I don't think there is any issue with DB connection.

  • reply

Anthony_Gourtay
November 15, 2019 - 9:01am

Could you please add details on your infra:
From what kind of server you run the upgrade, jdbc used, oracle DB version.
The DB user used for upgrade is having full access to DB?
Can you please add the config file for the upgrade also?
Thanks
Anthony

  • reply

AnandP
November 18, 2019 - 5:22am
Response to Anthony_Gourtay

Hi,
Below are the infra details you can refer to:--
Server is linux based operating system (Red Hat Enterpise v7)
App Server - Webpsphere v 8.5.5.16
Java 8
Oracle 12c and we have full admin access as per upgrade guide
Driver - Ojbc8.jar

Also, Please find the attached configuration file for upgrade.

 

 

 

  • reply

MatheswaranR
November 18, 2019 - 4:28am
Response to AnandP

Hi AnandP,

       I had downloaded and preview the file it contains configuration, can u share the link for the jdbc driver JAR file for the postgres DB? 

  • reply

Anthony_Gourtay
November 18, 2019 - 3:58am

for a 8.2.1 upgrade, from Websphere 8.5 but Oracle 11, I used ojdb7.jar.
Is it something you can try? I've not check detailled requirements for Ora12 and PEGA8.3.1

If you're at the first step of the upgrade, you need to put your RULES schema for both data & rules

rules.schema.name=PEG30_APP_RULES

data.schema.name=PEG30_APP_RULES
 

if you want to be able to resume the upgrade, ie not restarting from scratch, you need to put
automatic.resume=true

PS: I think you should now change your DB password :)

  • reply

AnandP
November 18, 2019 - 5:32am

Hi Anthony,

Thanks for your reply. I have already tried previous upgrade on Pega 8.3 on oracel 12c and Ojdbc8 and it worked fine for me and the same is advised in pega platform support guide as well so I don't think they are incompatible.

Regarding your suggestion on changing the data schema name, i guess that sounds true for either Out-of-place upgrade or single schema environment but since we are performing in-place upgrade on split schema, i think we need to provide the schema name separately and the same is suggested as per platform upgrade guide.

Regarding resuming the upgrade process, yes i have tried that as well but unfortunately it didn't work for me. We have already upgraded pega successfully previously so it is interesting to see why it's failing this time. My first guess is to look at the DB level as the log provides some indication in the same area but not sure where should I look it. If you have any specific question for DB, you can let me know and I can convey the same to our DB team.

Lastly, thanks for the advising on changing the DB password so I have removed the configuration file, however I was less worried as it was a POC environment and can only be accessed within our organization's network..:-)  

 

  • reply

Anthony_Gourtay
November 18, 2019 - 7:35am

in-place or out-place, global process is the same.

for in-place, you just don't need to prepare a new rule schema.

then, for step 1 of upgrade, via command line, the property file should have rule schema name for both data & rule variable. If you let empty the data schema viariable, then rule schema name is considered.

I checked 8.2.1 upgrade guide and it's well mentioned p 42 :)

If you use the UI, then indeed, data schema viariable should have data schema name.

Regards

Anthony

  • reply

AnandP
November 18, 2019 - 7:52am
Response to Anthony_Gourtay

Hi,

I was just wondering if we do not specify data schema name during in-palce upgrade then how pega scripts will identify the data schema to upgrade it because my impression was that upgrade scripts take care of both Rule & data schema upgrade if we specify their name separately in configuration file?

  • reply

Anthony_Gourtay
November 18, 2019 - 8:08am

The first upgrade.sh will take care of upgrading the rules.
You'll have then to run upgrade.sh --DataOnly which will also handle the data schema

 

  • reply

AnandP
November 18, 2019 - 8:40am

You may want to check the below community post where we discussed the same thing and it was advised that running upgrade.sh will take care of upgrading both rule & data schema if we provide them separately so need to run the upgrade.sh again with --dataOnly parameter

https://community1.pega.com/community/pega-support/question/place-upgrade-82

dataOnly parameter is only needed when you perform out of place upgrade and then migrate rule schema separately for other environments, then to upgrade data schema you need this additional parameters.

 

  • reply

Anthony_Gourtay
November 18, 2019 - 8:57am

but for now, it's your migration which fails :)
So it's worth a run with parameters as I mentioned or let's wait for someone else to give advice for your issue.
Regards
Anthony

  • reply

AnandP
November 26, 2019 - 5:06am

We tried upgrading on another database and there it worked so now we are checking with our DB team if anything is wrong with the previous database.

  • reply

AnandP
November 28, 2019 - 1:17pm

Hi,

This was a data issue. In our database, a corrupt/invalid instance of Data-Admin-DB-Name was found with all records as null, however the same was not visible within Pega. Since during upgrade pega validates all DB instances, this instance was causing the NullPointerException.

After deleting the record from database, upgrade started working.

  • reply