initial project release

.gitignore

@@ -0,0 +1,2 @@
+databunker
+bql.db

README.md

@@ -0,0 +1,768 @@
+# Paranoid Guy Data Bunker
+
+
+**Data Bunker is advanced personal information tokenization and storage service build to comply with GDPR.**
+
+This project, when deployed, can replace all user personal records scattered in the organization's different
+internal databases with one user token generated and managed by Data Bunker service.
+
+By deploying this project and moving all personal information to one place, you will comply with the following
+GDPR statement: *Personal data should be processed in a manner that ensures appropriate security and 
+confidentiality of the  personal data, including for preventing unauthorized access to or use of personal
+data and the equipment used for the processing.*
+
+**NOTE**: Implementing this project does not make you fully compliant with GDPR requirements and you still
+need to consult with an attorney specializing in privacy.
+
+#### Diagram of old-style solution.
+
+![picture](images/old-style-solution.png)
+
+#### Diagram of Solution with Paranoid Guy Data Bunker
+![picture](images/new-style-solution.png)
+
+
+---
+
+# This product stands many GDPR requirements
+
+## Right to be forgotten / Right to erasure
+
+When your customer asks for his **right to be forgotten** legal right, his private records will be
+wiped out of the Data Bunker database, giving you the possibility to leave all internal databases unchanged.
+
+**NOTE**: You just need to make sure that you do not have any user identifiable information in your other databases,
+logs, files, etc...
+
+## Right of access
+
+We build in passwordless login into the data bunker service. So, your customer/user can log in into his personal account
+at Data Bunker and view all information collected by Data Bunker in connection to his profile.
+
+## Right to rectification
+
+Your customer/user can log in to his personal account at Data Bunker and change his records. If needed, Bunker will
+send you a notification request about the change.
+
+## Right to restrict processing / Right to object
+
+Data Bunker can work as management for all user consents. User can cancel specific consent in his personal account at
+Bunker, for example, to block sending him emails. Your backend can work with Data Bunker using API to add, or cancel
+consents and we will send you a notification about user actions.
+
+## Right to data portability (partial)
+
+Your customer/user can log in to his personal account at Data Bunker and view and extract all his records stored at
+Data Bunker.
+
+**NOTE**: You need to provide your customers with a way to extract data from your internal databases.
+
+## Data minimisation
+
+Basically, when you clean up your databases from personal records and use Data Bunker token instead, you
+are already minimizing the personal information you store in different systems. In addition, when sending
+you customer data to 3rd party systems Data Bunker provides you with purposely build *shareable identity*
+that is time-bound.
+
+## Data Accuracy
+
+We allow the customer to change the records that are stored in Data Bunker. This way we achieve data accuracy.
+
+## Transparency
+
+All operations with personal records are saved in the audit log. Your customer can log in to his account at Data Bunker
+and view the audit trail.
+
+## Integrity and confidentiality
+
+All personal data is encrypted. Only relevant personnel can access the data. We audit all operations with personal records.
+All-access to Data Bunker API is done using an HTTPS SSL certificate. Enterprise version supports Shamir's Secret Sharing
+algorithm to split the master key to a number of keys. A number of keys (that can be saved in different hands in the
+organization) are required to bring up the system.
+
+## Accountability principle
+
+Each one, connected to Data Bunker must provide an access token to do any operation in Data Bunker or the user needs to
+login to access his own account. All operations are saved in the audit log.
+
+## Privacy by design
+
+This product, from the architecture level was build to comply with strick privacy laws. Deploying this or similar
+architecture, can make your company privacy by design compliant.
+
+## NOTE
+
+Implementing this project does not make you fully compliant with GDPR requirements and you still need to
+consult with an attorney specializing in privacy.
+
+---
+
+# Data Bunker usecases
+
+## Personal Information tokenization and storage
+
+This is already covered deeply above. Here I can add that Data Bunker has a layer of application
+level personal information storage and each user in our database can be linked to a number of
+application records (saved in Data Bunker).
+
+## Audit of all operations with personal records
+
+This is already covered above.
+
+## GDPR compliant logging
+
+Data Bunker supports a number of API that can help you to store user information in logs in
+GDPR compliant way and work with cloud logging companies.
+
+## Consent management, i.e. withdawal
+
+According to GDPR, if you want to send your customer SMS using 3rd party gateway,
+you must show to your customer a detailed notification message that you will send
+his phone number to a specific SMS gateway company and the user needs to confirm that.
+
+You need to store these confirmations and Data Bunker can help you with that.
+
+Consent must be freely given, specific, informed and unambiguous. From GDPR, Article 7, item 3:
+
+* **The data subject shall have the right to withdraw his or her consent at any time.**
+* **It shall be as easy to withdraw as to give consent.**
+
+In Data Bunker:
+
+* Your customers can log in to his Data Bunker account and view all consents he gave.
+* Users can also discharge consents and we will send you a notification message.
+
+
+## User signup and sign-in
+
+When implementing signup and sign-in in your customer-facing applications, we recommend you to
+store all signup records in the Data Bunker database. We support 3 types of indexes, index
+by login, index by email and index by phone. So you can easily implement login logic with
+our service.
+
+Index by email and index by phone allow us to give your customers passwordless access to their
+personal profile at Data Bunker. We send your user a one-time login code by SMS or email to
+give him access to his account at Data Bunker.
+
+
+---
+
+# Questions
+
+## Why Open Source?
+
+I am a big fan of the open-source movement. After a lot of thoughts and consultations,
+the main Data Bunker product will be open source.
+
+We are doing this to give our customers a steady base to continue using this solution in case
+the company is closed.
+
+Enterprise version will be closed source.
+
+## What is considered PII?
+
+Following it a partial list.
+
+* Name
+* Address
+* IP address
+* Browsing history
+* Political opinion
+* Sexual orientation
+* Social Security Number
+* Financial data
+* Banking data
+* Cookie data
+* Contacts
+* Mobile device ID
+* Passport data
+* Driving license
+* ID number
+* Health / medical data
+* RFID
+* Genetic info
+* Ethnic and racial information
+
+## Technology stack?
+
+I am a big fan of go language and I use it extensively.
+
+
+## Project technical features:
+
+* [Encrypted storage for personal information](#personal-information-tokanization)
+* [Application data separation](#application-data-separation)
+* [Time-limited passwordless access to personal information](#time-limited-passwordless-access-to-personal-information)
+* [Web and mobile app session data storage](#web-and-mobile-app-session-data-storage)
+* [Time-limited passwordless access to web and app session data](#web-and-mobile-app-session-data-storage)
+* [Share user identiy with 3rd party services](#share-user-identity-with-3rd-parties)
+* [User consent management, storage & withdrawal](#user-consent-management)
+* [Audit of all operations](#audit)
+* [Customer UI](#user-ui)
+* [User passwordless authentication](#custom-user-index)
+
+## Enterprise features
+* [Split master key with Shamir's Secret Sharing algo](#master-key-split-in-enterprise-version)
+* [Advaned role management, ACL](#advanced-acl)
+* [Support Hashicorp Vault](#hashicorp-vault-integration)
+
+---
+
+## Encryption in motion and encryption in storage
+
+All access to Data Bunker API is done using HTTPS SSL certificate. All records that have user personal information
+are encrypted or securely hashed in the databases. All user records are encrypted with a 32 byte key comprizing of
+System Master key (24 bytes, stored in memory, not on disk) and user record key (8 bytes, stored on disk).
+
+### Master key split in Enterprise version
+
+Upon initial start, the **Enterprise version** generates a secret master key and 5 keys out of it.
+These 5 keys are generated using Shamir's Secret Sharing algorithm. Combining 3 of any of the keys,
+ejects original master key and that can be used to decrypt all records.
+
+The Master key is kept in RAM and is never stored to disk. You will need to provide 3 kits to unlock the application.
+It is possible to save these keys in the AWS secret store and other vault services.
+
+---
+
+## Data Bunker internal tables
+
+Information inside Data Bunker is saved in multiple tables in encrypted format. Here is a diagram of tables.
+
+Detailed usecase for each table is covered bellow.
+
+
+![picture](images/data-bunker-tables.png)
+
+---
+
+## Personal information tokanization
+
+User information, or PII, received in HTML POST key/value format of or JSON format is serialized, encrypted
+with a 32 byte key and saved in database. You will get a user token to use in internal databases. Afterwords,
+you can query the Data Bunker service to receive personal information, saving audit trail.
+
+![picture](images/create-user-token-flow.png)
+
+---
+
+## Application data separation
+
+When creating application, I suppose you do not want to mix your customer data with data from other applications.
+In addition to personal information record, Data Bunker provides you a way to store your app user information in a
+specific type of record for that. So, you can retreave only your app' user personal information.
+
+![picture](images/create-user-app-record.png)
+
+---
+
+## Web and mobile app session data storage
+
+Web or mobile application session data is very similar. They contain customer IP address, browser information,
+web server headers, logged-in user info, etc... Many systems, including popular webservers, like Nginx, Apache
+simply store this information in logs. This information, according to GDPR is considered personal identifiable
+information and must be secured and controlled.
+
+So, you can not save user ip or browser information in logs now. Insead, Data Bunker will generate you a token to
+save in logs. Data Bunker provides you an API to retreave this info out of Data Bunker without additional password
+for a limited time as in GDPR. For example one month.
+
+![picture](images/create-user-session-flow.png)
+
+---
+
+## Time-limited passwordless access to personal information
+
+Sometimes you want to share user, app or session private information in less trusted systems without providing
+access to system root token.
+
+Data Bunker has an API that allows you to generate temprorary access token to access specific fields in the
+user personal record or application level data or a session record for a limited time only.
+
+Your partner can retrieve this information and only specific fields during this specific timeframe.
+Afterward, access will be blocked.
+
+**IMAGE**
+
+---
+
+## Shareable user identity for 3rd parties
+
+When sharing data with 3rd party services like web analytics, logging, intelligence, etc... sometimes we need to
+share user id, for example, customer original IP address or email address. All these pieces of information
+are considred user identifiable information and must be minimized when sending to 3rd paty systems.
+
+***Do not share your customer user name, IP, emails, etc... because they look nice in reports!***
+
+According to GDPR: *The personal data should be adequate, relevant and **limited to what is necessary** for the
+purposes for which they are processed.*
+
+Our system can generate you time-limited shareable identity token that you can share with 3rd parties as an identity.
+This identity, can link bacck to the user personal record or user app record or to specific user session.
+
+Optionally, Data Bunker can incorporate partner name in identity so, you track this identity usage.
+
+**IMAGE**
+
+---
+
+## User consent management
+
+Consent in GDPR terms is clear approval for example to share user information with 3rd party, for example with SMS
+gateway company to send him urgent notifications.
+
+Consent must be freely given, specific, informed and unambiguous. From GDPR, Article 7, item 3:
+
+* **The data subject shall have the right to withdraw his or her consent at any time.**
+* **It shall be as easy to withdraw as to give consent.**
+
+To comply with this requirement, we support storage and management of user consent by API level and in user UI.
+
+---
+
+## Audit
+
+Data Bunker saves audit events on all API operation. For example, new personal record added or changed; personal information
+record retreaved, etc...
+
+By providing Audit of events, in relation to personal data, provides response to GDRP Article 15 requirement:
+*Right of access by the data subject*.
+
+Special features:
+
+* Personal information in audit event is encrypted.
+* User can view his own records only.
+
+Each audit record consists of:
+
+* Date and time
+* Operation title
+* Operation status
+* Operation description
+* Change before and after if applicable
+* User session info if available: IP address, headers, etc...
+
+**IMAGE**
+
+Example from google: https://console.cloud.google.com/home/activity
+
+---
+
+## User UI
+
+Internal UI is build to allow users to login with their email and access all data collected by this system.
+You can easily change it to your requirements.
+
+According to GDPR, controller must provide Data subject with:
+
+* Right of access by the data subject (Article 15)
+* Right to rectification (Article 16)
+* Right to be forgotten (Article 17)
+* Right to restriction of processing (Article 18)
+* Notification obligation regarding rectification or erasure of personal data or restriction of processing (Article 19)
+* Right to data portability (Article 20)
+* Right to object (Article 21)
+
+---
+
+# Enterprise features
+
+## Advanced role management, ACL
+
+By default, all access to Data Bunker is done with one root token or with **Time-limited passwordless access tokens**
+that allow to read data from specific user record only.
+
+For more granular control, Data Bunker supports the notion of custom roles. For example, you can create a role
+to view all records or another role to add and change any user records; view sessions, view all audit events, etc...
+
+After you define a role, the system allow you to generate access token for this role (you will need to have root token
+for all these operations).
+
+Data Bunker have an API for all these operations.
+
+## Support Hashicorp Vault
+
+Hashicorp Vault, is a great piece of new generation of security product, has a notion of session accounts/passwords.
+Hashicorp Vault can store root access token to Paranoid Guy Data Bunker, and when your application wants to open
+session and access Data Bunker, it will talk with Bunker to issue a temp token with specified role.
+When your application session is closed with Data Bunker, Hashicorp Vault will connect to Data Bunker and revoke access token.
+
+This architecture is done to minimize the chance that if the attacker breakes into your application server,
+he will not get a full controll over the Data Bunker service as root token will not be saved in your
+application server.
+
+This is all done with the help of custom plugin we build for Hashicorp Vault.
+
+---
+
+## User Api
+
+
+| Resource / HTTP method | POST (create)    | GET (read)    | PUT (update)     | DELETE (delete)  |
+| ---------------------- | ---------------- | ------------- | ---------------- | ---------------- |
+| /v1/user               | Create new user  | Error         | Error            | Error            |
+| /v1/user/uuid/{token}  | Error            | Get user      | Update user      | Delete user PII  |
+| /v1/user/login/{login} | Error            | Get user      | Update user      | Delete user PII  |
+| /v1/user/email/{email} | Error            | Get user      | Update user      | Delete user PII  |
+| /v1/user/phone/{phone} | Error            | Get user      | Update user      | Delete user PII  |
+
+
+## Create user record
+### `POST /v1/user`
+
+### Explanation
+
+This API is used to create new user record and if the request is successful it returns new `{token}`.
+On the database level, each records is encrypted with it's own key.
+
+
+### POST Body Format
+
+POST Body can contain regular form data or JSON. Data Bunker extracts `{login}`, `{phoen}` and `{email}` out of
+POST data or from JSON first level and builds additional hashed index for user object. These fields, if
+provided must be unique, otherwise you will ge an error. So, you can not create additional user object
+with duplicate email.
+
+The following content type supported:
+
+* **application/json**
+* **application/x-www-form-urlencoded**
+
+
+### Example:
+
+Create used by posting JSON:
+
+```
+curl -s http://localhost:3000/v1/user -XPOST \
+  -H "X-Bunker-Token: cb2537f9-14e2-7019-503f-b36a1a8f6e7f" \
+  -H "Content-Type: application/json" \
+  -d '{"firstName": "John","lastName":"Doe","email":"user@gmail.com"}'
+{"status":"ok","uuid":"db80789b-0ad7-0690-035a-fd2c42531e87"}
+```
+
+Create user by POSTing user key/value fiels as post parameters:
+
+```
+curl -s http://localhost:3000/v1/user -XPOST \
+  -H "X-Bunker-Token: $TOKEN" \
+  -d 'firstName=John' \
+  -d 'lastName=Doe' \
+  -d 'email=user2@gmail.com'
+{"status":"ok","uuid":"db80789b-0ad7-0690-035a-fd2c42531e87"}
+```
+
+**NOTE**: Keep this user token privately as it provides user private information in your system.
+For semi-trusted environments or 3rd party companies, use **shareable identity** instead.
+
+
+---
+
+## Get user record
+### `GET /v1/user/{uuid,login,email,phone}/{indexValue}`
+
+### Explanation
+This API is used to get user PII records. You can lookup user token by **uuid** (token), **email**, **phone** or **login**.
+
+### Example:
+
+Fetch by user token:
+
+```
+curl --header "X-Bunker-Token: $TOKEN" -XGET \
+   https://localhost:3000/v1/user/uuid/DAD2474A-E9A7-4BA7-BFC2-C4506880198E
+{"uuid":"DAD2474A-E9A7-4BA7-BFC2-C4506880198E","data":{"k1":[1,10,20],
+"k2":{"f1":"t1","f3":{"a":"b"}},"login":"user1","name":"tom"}}
+```
+
+Fetch by "login" name:
+
+```
+curl --header "X-Bunker-Token: $TOKEN" -XGET \
+   https://localhost:3000/v1/user/login/user1
+{"uuid":"DAD2474A-E9A7-4BA7-BFC2-C4506880198E","data":{"k1":[1,10,20],
+"k2":{"f1":"t1","f3":{"a":"b"}},"login":"user1","name":"tom"}}
+```
+
+
+---
+
+## Update user record
+### `PUT /v1/user/{uuid,login,email,phone}/{indexValue}`
+
+### Explanation
+
+This API is used to update user record. You can update user by **uuid** (token), **email**, **phone** or **login**.
+This call returns update status on success or error message on error.
+
+### POST Body Format
+
+POST Body can contain regular form POST data or JSON. When using JSON, you can remove the record by setting it's value to null.
+For example {"key-to-delete":null}.
+
+The following content type supported:
+
+* **application/json**
+* **application/x-www-form-urlencoded**
+
+
+### Example:
+
+The following command will change user name to "Alex". An audit event will be generated showing previous and new value.
+
+```
+curl --header "X-Bunker-Token: $TOKEN" -d 'name=Alex' -XPUT \
+   https://localhost:3000/v1/user/uuid/DAD2474A-E9A7-4BA7-BFC2-C4506880198E
+```
+
+---
+
+## Delete user by record
+### `DELETE /v1/user/{uuid,login,email,phone}/{indexValue}`
+
+This command will remove all user records from the database, leaving only user token id.
+
+```
+curl -header "X-Bunker-Token: $TOKEN" -XDELETE \
+  https://localhost:3000/v1/user/uuid/DAD2474A-E9A7-4BA7-BFC2-C4506880198E
+{"status":"ok","result":"done"}
+```
+
+## User App Api
+
+
+| Resource / HTTP method          | POST (create)       | GET (read)            | PUT (update)  | DELETE |
+| ------------------------------- | ------------------- | --------------------- | ------------- | ------ |
+| /v1/userapp/uuid/:uuid/:appname | Create new user app | Get record            | Change record | Delete |
+| /v1/userapp/uuid/:uuid          | Error               | Get all user app list | Error         | Error  |
+| /v1/userapp/list                | Error               | Get all app list      | Error         | Error  |
+
+
+## Create user app record
+### `POST /v1/userapp/uuid/:uuid/:appname`
+
+### Explanation
+
+This API is used to create new user app record and if the request is successful it returns new `{token}`.
+
+
+---
+
+## User Session Api
+
+| Resource / HTTP method    | POST (create)      | GET (read)     | PUT (update)   | DELETE (delete) |
+| ------------------------- | ------------------ | -------------- | -------------- | --------------- |
+| /v1/session/uuid/:uuid    | Create new session | Get sessions   | Error          | Error           |
+| /v1/session/session/:uuid | Error              | Get session    | Error??        | Error??         |
+| /v1/session/clientip/:ip  | Error              | Get sessions   | Error          | Error           |
+
+## Create user session record
+### `POST /v1/session/uuid/:uuid`
+
+### Explanation
+
+This API is used to create new user session and if the request is successful it returns new `{session}`.
+You can now use this id in your logs instead of user IP and browser user-agent info, etc...
+
+Our API supports generation of session tokens based on the following information:
+user ip, mobile device info, user agent, etc...
+
+You can send the data as JSON POST or as regular POST parameters when working with this API.
+
+## Get user session record
+### `GET /v1/session/session/:session`
+
+### Explanation
+
+This API returns session data.
+
+
+## Get session records by user token.
+### `GET /v1/session/uuid/:session`
+
+### Explanation
+
+This API returns an array of sessions of the same user.
+
+## Get session records by ip address.
+### `GET /v1/session/clientip/:session`
+
+### Explanation
+
+This API returns an array of user sessions by IP address. These sessions can be of different people.
+
+---
+
+## Passwordless tokens API
+
+| Resource / HTTP method | POST (create)     | GET (read)    | PUT (update)     | DELETE (delete)  |
+| ---------------------- | ----------------- | ------------- | ---------------- | ---------------- |
+| /v1/token/:uuid        | Create new record | Error         | Error            | Error            |
+| /v1/token/:token       | Error             | Get data      | Error            | Error            |
+
+	router.POST("/v1/token/:token", e.userNewToken)
+	router.GET("/v1/token/:xtoken", e.userCheckToken)
+
+
+---
+
+## Shareable token API
+
+| Resource / HTTP method | POST (create)     | GET (read)    | PUT (update)     | DELETE (delete)  |
+| ---------------------- | ----------------- | ------------- | ---------------- | ---------------- |
+| /v1/shareable/:uuid    | Create new record | Error         | Error            | Error            |
+| /v1/shareable/:token   | Error             | Get data      | Error            | Error            |
+
+
+---
+
+**TODO-FINISH**
+
+
+## Temporary user access tokens
+
+Sometimes, for example, when working with 3rd party partners or semi-trusted environments, you might
+need to generate a user access token with a specific expiration time. Your partner can retrieve user
+information during this specific time only.
+Afterward, access will be blocked.
+
+The following command will generate a token to access user email and name for 7 days:
+
+```
+curl --header "X-Bunker-Token: $TOKEN" -d 'fields=email,name' -d 'expiration=7d' -d 'partner=sms' \
+   https://bunker.company.com/gentokens/DAD2474A-E9A7-4BA7-BFC2-C4506880198E
+```
+
+Output:
+
+```
+476E41E7-72AD-448A-BB43-7ACDB8C53735
+```
+
+
+### 3rd party logging
+
+Instead of maintaining internal logs, a lot of companies are using 3rd party logging facility like logz or coralogix or something else.
+To improve adherence to GDPR, we build a special feature - generate specific session id for such 3rd party service.
+
+When using these uuids in external systems, you basically **pseudonymise personal data**. In addition, in accordance with GDPR Article 5:
+**Principles relating to processing of personal data**. Personal data shall be: (c) 
+adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘**data minimisation**’);
+
+Here is a command to do it:
+
+```
+curl -d 'ip=user@example.com' \
+     -d 'user-agent=mozila' \
+     -d 'partner=coralogix' \
+     -d 'expiration=7d'\
+   https://bunker.company.com/gensession/DAD2474A-E9A7-4BA7-BFC2-C4506880198E
+```
+
+It will generate a new uuid, that you can now pass to 3rd party system as a user id.
+
+
+## User consent management
+
+One of the GDPR requirements is the storage of user consent. For example, your customer must approve to receive email marketing information.
+
+Using the GDPR language, your customer must give explicit consent to receive marketing information.
+
+Consent must be freely given, specific, informed and unambiguous. From GDPR, Article 7, item 3:
+
+* **The data subject shall have the right to withdraw his or her consent at any time.**
+* **It shall be as easy to withdraw as to give consent.**
+
+To comply with this requirement, we added support to manage user consent. We support the following APIs:
+
+
+### List granted
+
+```
+curl --header "X-Bunker-Token: $TOKEN" \
+   https://bunker.company.com/consent/DAD2474A-E9A7-4BA7-BFC2-C4506880198E
+```
+
+### List all
+
+```
+curl --header "X-Bunker-Token: $TOKEN" \
+   https://bunker.company.com/consent/DAD2474A-E9A7-4BA7-BFC2-C4506880198E?all
+```
+
+### Cancel consent
+
+```
+curl --header "X-Bunker-Token: $TOKEN" -XDELETE \
+   https://bunker.company.com/consent/DAD2474A-E9A7-4BA7-BFC2-C4506880198E/<consent-id>
+```
+
+### User gives consent
+
+**TODO**
+
+### Easily cancel consent for email marketing
+
+For example, for email marketing, users got distracted, when they need to login in order to unsubscribe from the newsletter.
+To simplify this operation, users will be allowed to unsubscribe only using email address without full login operation.
+
+### Unlock bunker
+
+Run the following command with different keys:
+
+```
+bunker unlock **key**
+```
+
+Or you can provide multiple keys at once:
+
+```
+bunker unlock key1 key2 key3
+```
+
+### View lock status
+
+```
+bunker status | jq .lock
+```
+
+Result:
+
+```
+locked
+```
+
+
+## Audit API
+
+
+
+
+It is not compliant, unless you have a real reason to share this specific personal sub-record. For example,
+sending customer phone when notifying customer using 3rd party SMS gateway.
+
+
+
+# SECTION IS NOT UPDATED BELLOW
+
+## Data Bunker init
+
+Upon initial init, the Data Bunker service will check if the system is initialized for the first time, and if yes,
+it will generate root password, master key and derived keys out of it. Otherwise, an error will be printed.
+
+```
+bunker init
+```
+
+Output:
+
+```
+Root password: 123456
+Key1: abcdefg
+Key2: abcdefg
+key3: abcdefg
+Key4: abcdefg
+Key5: abcdefg
+```
+
+**TODO**: Secret keys printed to output can be easily extracted in cloud environments for example in Kubernetes logs!

build.sh

@@ -0,0 +1,10 @@
+# build without debug
+go build -ldflags "-w" -o databunker ./src/bunker.go ./src/qldb.go ./src/audit_db.go ./src/audit_api.go \
+  ./src/utils.go ./src/cryptor.go \
+  ./src/sms.go ./src/email.go \
+  ./src/users_db.go ./src/users_api.go \
+  ./src/userapps_db.go ./src/userapps_api.go \
+  ./src/sessions_db.go \
+  ./src/consent_db.go ./src/consent_api.go \
+  ./src/xtokens_db.go ./src/xtokens_api.go
+

databunker.yaml

@@ -0,0 +1,20 @@
+# Server configurations
+generic:
+  # allow to create user object without login
+  create_user_without_token: true
+  #notification_url: "http://localhost/"
+sms:
+  # default country when sending out SMSM
+  twilio_account: "AC"
+  twilio_token: "af"
+  twilio_from: "+180"
+  default_country: "UK"
+server:
+  host: "localhost"
+  port: 3000
+smtp:
+  server: "smtp.eu.mailgun.org"
+  port: 587
+  user: "postmaster@mg.your-company.com"
+  pass: "b6"
+  sender: "botdatabunker.your-company.com"

src/audit_api.go

@@ -0,0 +1,42 @@
+package main
+
+import (
+	"fmt"
+	"net/http"
+
+	"github.com/julienschmidt/httprouter"
+)
+
+func (e mainEnv) getAuditEvents(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	userTOKEN := ps.ByName("token")
+	event := audit("view audit events", userTOKEN)
+	defer func() { event.submit(e.db) }()
+	//fmt.Println("error code")
+	if enforceUUID(w, userTOKEN, event) == false {
+		return
+	}
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	var offset int32
+	var limit int32 = 10
+	args := r.URL.Query()
+	if value, ok := args["offset"]; ok {
+		offset = atoi(value[0])
+	}
+	if value, ok := args["limit"]; ok {
+		limit = atoi(value[0])
+	}
+	resultJSON, counter, err := e.db.getAuditEvents(userTOKEN, offset, limit)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	fmt.Printf("Total count of events: %d\n", counter)
+	//fmt.Fprintf(w, "<html><head><title>title</title></head>")
+	w.Header().Set("Access-Control-Allow-Origin", "*")
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	str := fmt.Sprintf(`{"total":%d,"rows":%s}`, counter, resultJSON)
+	w.Write([]byte(str))
+}

src/audit_db.go

@@ -0,0 +1,98 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"time"
+
+	"go.mongodb.org/mongo-driver/bson"
+)
+
+type auditEvent struct {
+	When   int32  `json:"when"`
+	Who    string `json:"who"`
+	Record string `json:"record"`
+	App    string `json:"app"`
+	Title  string `json:"title"`
+	Status string `json:"status"`
+	Msg    string `json:"msg"`
+	Debug  string `json:"debug"`
+	Before string `json:"before"`
+	After  string `json:"after"`
+	Meta   string `json:"meta"`
+}
+
+func audit(title string, record string) *auditEvent {
+	fmt.Printf("/%s : %s\n", title, record)
+	return &auditEvent{Title: title, Record: record, Status: "ok", When: int32(time.Now().Unix())}
+}
+
+func auditApp(title string, record string, app string) *auditEvent {
+	fmt.Printf("/%s : %s : %s\n", title, app, record)
+	return &auditEvent{Title: title, Record: record, Status: "ok", When: int32(time.Now().Unix())}
+}
+
+func (event auditEvent) submit(db dbcon) {
+	//fmt.Println("submit event to audit!!!!!!!!!!")
+	/*
+		bdoc, err := bson.Marshal(event)
+		if err != nil {
+			fmt.Printf("failed to marshal audit event: %s\n", err)
+			return
+		}
+		var bdoc2 bson.M
+		err = bson.Unmarshal(bdoc, &bdoc2)
+		if err != nil {
+			fmt.Printf("failed to marshal audit event2: %s\n", err)
+			return
+		}*/
+	bdoc := bson.M{}
+	bdoc["when"] = event.When
+	if len(event.Who) > 0 {
+		bdoc["who"] = event.Who
+	}
+	if len(event.Record) > 0 {
+		bdoc["record"] = event.Record
+	}
+	if len(event.App) > 0 {
+		bdoc["app"] = event.App
+	}
+	if len(event.Title) > 0 {
+		bdoc["title"] = event.Title
+	}
+	bdoc["status"] = event.Status
+	if len(event.Msg) > 0 {
+		bdoc["msg"] = event.Msg
+	}
+	if len(event.Debug) > 0 {
+		bdoc["debug"] = event.Debug
+	}
+	if len(event.Before) > 0 {
+		bdoc["before"] = event.Before
+	}
+	if len(event.After) > 0 {
+		bdoc["after"] = event.After
+	}
+	if len(event.Meta) > 0 {
+		bdoc["meta"] = event.Meta
+	}
+	_, err := db.createRecord(TblName.Audit, &bdoc)
+	//_, err := db.audit.InsertOne(context.TODO(), &bdoc)
+	if err != nil {
+		fmt.Printf("failed to marshal audit event: %s\n", err)
+		return
+	}
+	//fmt.Printf("done!!!")
+}
+
+func (dbobj dbcon) getAuditEvents(userTOKEN string, offset int32, limit int32) ([]byte, int64, error) {
+	//var results []*auditEvent
+	count, err := dbobj.countRecords(TblName.Audit, "record", userTOKEN)
+	if err != nil {
+		return nil, 0, err
+	}
+	records, err := dbobj.getList(TblName.Audit, "record", userTOKEN, offset, limit)
+	resultJSON, err := json.Marshal(records)
+	//fmt.Printf("Found multiple documents (array of pointers): %+v\n", results)
+	return resultJSON, count, nil
+}

src/bunker.go

@@ -0,0 +1,255 @@
+package main
+
+import (
+	"encoding/hex"
+	"flag"
+	"fmt"
+	"log"
+	"math/rand"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/gobuffalo/packr"
+	"github.com/julienschmidt/httprouter"
+	"github.com/kelseyhightower/envconfig"
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promhttp"
+	yaml "gopkg.in/yaml.v2"
+)
+
+type Tbl = int
+
+type listTbls struct {
+	Users    Tbl
+	Audit    Tbl
+	Xtokens  Tbl
+	Consent  Tbl
+	Sessions Tbl
+}
+
+// Enum for public use
+var TblName = &listTbls{
+	Users:    0,
+	Audit:    1,
+	Xtokens:  2,
+	Consent:  3,
+	Sessions: 4,
+}
+
+type Config struct {
+	Generic struct {
+		Create_user_without_token bool `yaml:"create_user_without_token"`
+	}
+	Sms struct {
+		Default_country string `yaml:"default_country"`
+		Twilio_account  string `yaml:"twilio_account"`
+		Twilio_token    string `yaml:"twilio_token"`
+		Twilio_from     string `yaml:"twilio_from"`
+	}
+	Server struct {
+		Port string `yaml:"port", envconfig:"BUNKER_PORT"`
+		Host string `yaml:"host", envconfig:"BUNKER_HOST"`
+	} `yaml:"server"`
+	Smtp struct {
+		Server string `yaml:"server", envconfig:"SMTP_SERVER"`
+		Port   string `yaml:"port", envconfig:"SMTP_PORT"`
+		User   string `yaml:"user", envconfig:"SMTP_USER"`
+		Pass   string `yaml:"pass", envconfig:"SMTP_PASS"`
+		Sender string `yaml:"sender", envconfig:"SMTP_SENDER"`
+	} `yaml:"smtp"`
+}
+
+type mainEnv struct {
+	db   dbcon
+	conf Config
+}
+
+type userJSON struct {
+	jsonData []byte
+	loginIdx string
+	emailIdx string
+	phoneIdx string
+}
+
+type tokenAuthResult struct {
+	ttype   string
+	name    string
+	token   string
+	fields  string
+	appName string
+}
+
+func prometheusHandler() http.Handler {
+	handlerOptions := promhttp.HandlerOpts{
+		ErrorHandling:      promhttp.ContinueOnError,
+		DisableCompression: true,
+	}
+	promHandler := promhttp.HandlerFor(prometheus.DefaultGatherer, handlerOptions)
+	return promHandler
+}
+
+func (e mainEnv) metrics(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	fmt.Printf("/metrics\n")
+	//w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	//fmt.Fprintf(w, `{"status":"ok","apps":%q}`, result)
+	//fmt.Fprintf(w, "hello")
+	//promhttp.Handler().ServeHTTP(w, r)
+	prometheusHandler().ServeHTTP(w, r)
+}
+
+func (e mainEnv) index(w http.ResponseWriter, r *http.Request) {
+	fmt.Printf("Index access\n")
+	/*
+		if r.Method != "GET" {
+			http.Error(w, http.StatusText(405), 405)
+			log.Panic("Method %s", r.Method)
+			return
+		}
+	*/
+	fmt.Fprintf(w, "<html><head><title>title</title></head></html>")
+}
+
+func (e mainEnv) setupRouter() *httprouter.Router {
+
+	box := packr.NewBox("../ui")
+
+	router := httprouter.New()
+	router.POST("/v1/user", e.userNew)
+	router.GET("/v1/user/:index/:code", e.userGet)
+	router.DELETE("/v1/user/:index/:code", e.userDelete)
+	router.PUT("/v1/user/:index/:code", e.userChange)
+
+	router.GET("/v1/login/:index/:code", e.userLogin)
+	router.GET("/v1/enter/:index/:code/:tmp", e.userLoginEnter)
+
+	router.POST("/v1/xtoken/:token", e.userNewToken)
+	router.GET("/v1/xtoken/:xtoken", e.userCheckToken)
+
+	router.GET("/v1/consent/:index/:code", e.consentList)
+	router.POST("/v1/consent/:index/:code", e.consentAccept)
+	//router.PATCH("/v1/consent/:index/:code", e.consentCancel)
+	router.DELETE("/v1/consent/:index/:code", e.consentCancel)
+
+	router.POST("/v1/userapp/token/:token/:appname", e.userappNew)
+	router.GET("/v1/userapp/token/:token/:appname", e.userappGet)
+	router.PUT("/v1/userapp/token/:token/:appname", e.userappChange)
+	router.GET("/v1/userapp/token/:token", e.userappList)
+	router.GET("/v1/userapp/list", e.appList)
+
+	router.GET("/v1/metrics", e.metrics)
+
+	router.GET("/v1/audit/list/:token", e.getAuditEvents)
+
+	router.GET("/", func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+		data, err := box.Find("index.html")
+		if err != nil {
+			//log.Panic("error %s", err.Error())
+			fmt.Printf("404 %s, error: %s\n", r.URL.Path, err.Error())
+			w.WriteHeader(404)
+		} else {
+			//fmt.Printf("return static file: %s\n", data)
+			fmt.Printf("200 %s\n", r.URL.Path)
+			w.WriteHeader(200)
+			w.Write([]byte(data))
+		}
+	})
+	router.GET("/site/*filepath", func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+		data, err := box.Find(r.URL.Path)
+		if err != nil {
+			fmt.Printf("404 GET %s\n", r.URL.Path)
+			w.WriteHeader(404)
+		} else {
+			//w.Header().Set("Access-Control-Allow-Origin", "*")
+			if strings.HasSuffix(r.URL.Path, ".css") {
+				w.Header().Set("Content-Type", "text/css")
+			} else if strings.HasSuffix(r.URL.Path, ".js") {
+				w.Header().Set("Content-Type", "text/javascript")
+			}
+			// text/plain
+			fmt.Printf("200 %s\n", r.URL.Path)
+			w.WriteHeader(200)
+			w.Write([]byte(data))
+		}
+	})
+	router.NotFound = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
+		w.WriteHeader(http.StatusNotFound)
+		w.Write([]byte("url not found"))
+		fmt.Printf("404 %s %s\n", r.Method, r.URL.Path)
+	})
+	return router
+}
+
+func readFile(cfg *Config) error {
+	f, err := os.Open("databunker.yaml")
+	if err != nil {
+		return err
+	}
+	decoder := yaml.NewDecoder(f)
+	err = decoder.Decode(cfg)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+func readEnv(cfg *Config) error {
+	err := envconfig.Process("", cfg)
+	return err
+}
+
+func main() {
+	rand.Seed(time.Now().UnixNano())
+	fmt.Println("***MAIN***")
+	lockMemory()
+	var cfg Config
+	readFile(&cfg)
+	readEnv(&cfg)
+	fmt.Printf("%+v\n", cfg)
+	initPtr := flag.Bool("init", false, "a bool")
+	masterKeyPtr := flag.String("masterkey", "", "master key")
+	flag.Parse()
+	var err error
+	var masterKey []byte
+	if *initPtr {
+		fmt.Println("Init")
+		masterKey, err = generateMasterKey()
+		fmt.Printf("Master key: %x\n", masterKey)
+	} else if masterKeyPtr != nil && len(*masterKeyPtr) > 0 {
+		masterKey, err = hex.DecodeString(*masterKeyPtr)
+	} else {
+		fmt.Println("Run ./databunker -init for the firts time.")
+		log.Fatal("Masterkey is missing. Run ./databunker -masterkey key")
+	}
+	if err != nil {
+		//log.Panic("error %s", err.Error())
+		fmt.Printf("error %s", err.Error())
+	}
+	db, _ := newDB(masterKey, nil)
+	if *initPtr {
+		fmt.Println("Init")
+		db.initDB()
+		rootToken, err := db.createRootToken()
+		if err != nil {
+			//log.Panic("error %s", err.Error())
+			fmt.Printf("error %s", err.Error())
+		}
+		fmt.Printf("Root token: %s\n", rootToken)
+	}
+	db.initUserApps()
+	e := mainEnv{db, cfg}
+	fmt.Printf("host %s\n", cfg.Server.Host+":"+cfg.Server.Port)
+	router := e.setupRouter()
+	if _, err := os.Stat("./server.key"); !os.IsNotExist(err) {
+		//TODO
+		fmt.Printf("Loading ssl\n")
+		err := http.ListenAndServeTLS(":443", "server.ctr", "server.key", router)
+		if err != nil {
+			log.Fatal("ListenAndServe: ", err)
+		}
+	} else {
+		log.Fatal(http.ListenAndServe(cfg.Server.Host+":"+cfg.Server.Port, router))
+	}
+}

src/bunker_test.go

@@ -0,0 +1,115 @@
+package main
+
+import (
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/julienschmidt/httprouter"
+)
+
+func TestCreateAPIUser(t *testing.T) {
+	masterKey, _ := hex.DecodeString("71c65924336c5e6f41129b6f0540ad03d2a8bf7e9b10db72")
+	db, _ := newDB(masterKey, nil)
+	var cfg Config
+	e := mainEnv{db, cfg}
+
+	rootToken, err := e.db.getRootToken()
+	if err != nil {
+		t.Fatalf("Failed to retreave root token: %s\n", err)
+	}
+	userJSON := `{"login":"abcdefg","name":"tom","pass":"mylittlepony","k1":[1,10,20],"k2":{"f1":"t1","f3":{"a":"b"}},"admin":true}`
+
+	request := httptest.NewRequest("POST", "/user", strings.NewReader(userJSON))
+	request.Header.Set("Content-Type", "application/json")
+	request.Header.Set("X-Bunker-Token", rootToken)
+	//var resp http.ResponseWriter
+	rr := httptest.NewRecorder()
+	var ps httprouter.Params
+	e.userNew(rr, request, ps)
+
+	//fmt.Printf("After create user------------------\n%s\n\n\n", rr.Body)
+	var raw map[string]interface{}
+	err = json.Unmarshal(rr.Body.Bytes(), &raw)
+	if err != nil {
+		t.Fatalf("Failed to parse json response on user create: %s\n", err)
+	}
+	var userTOKEN string
+	if status, ok := raw["status"]; ok {
+		if status == "error" {
+			if strings.HasPrefix(raw["message"].(string), "duplicate") {
+				_, userTOKEN, _ = e.db.getUserIndex("abcdefg", "login")
+				fmt.Printf("user already exists: %s\n", userTOKEN)
+			} else {
+				t.Fatalf("Failed to create user: %s\n", raw["message"])
+				return
+			}
+		} else if status == "ok" {
+			userTOKEN = raw["token"].(string)
+		}
+	}
+	if len(userTOKEN) == 0 {
+		t.Fatalf("Failed to parse user UUID")
+	}
+	p2 := httprouter.Param{"token", userTOKEN}
+	ps2 := []httprouter.Param{p2}
+
+	pars := `{"expiration":"1d","fields":"uuid,name,pass,k1,k2.f3"}`
+	request = httptest.NewRequest("POST", "/user", strings.NewReader(pars))
+	request.Header.Set("Content-Type", "application/json")
+	request.Header.Set("X-Bunker-Token", rootToken)
+	//var resp http.ResponseWriter
+	rr = httptest.NewRecorder()
+	e.userNewToken(rr, request, ps2)
+	//fmt.Printf("after create token------------------\n%s\n\n\n", rr.Body)
+	err = json.Unmarshal(rr.Body.Bytes(), &raw)
+	if err != nil {
+		fmt.Printf("Failed to parse json response on user create: %s\n", err)
+	}
+	tokenUUID := ""
+	if status, ok := raw["status"]; ok {
+		if status == "error" {
+			t.Fatalf("Failed to create user token: %s\n", raw["message"])
+			return
+		} else if status == "ok" {
+			tokenUUID = raw["xtoken"].(string)
+		}
+	}
+	if len(tokenUUID) == 0 {
+		t.Fatalf("Failed to retreave user token: %s\n", rr.Body)
+	}
+	fmt.Printf("User token: %s\n", tokenUUID)
+
+	request = httptest.NewRequest("GET", "/user", nil)
+	request.Header.Set("Content-Type", "application/json")
+	request.Header.Set("X-Bunker-Token", rootToken)
+	//var resp http.ResponseWriter
+	rr = httptest.NewRecorder()
+
+	p3 := httprouter.Param{"xtoken", tokenUUID}
+	ps3 := []httprouter.Param{p3}
+	e.userCheckToken(rr, request, ps3)
+	fmt.Printf("get by token------------------\n%s\n\n\n", rr.Body)
+	err = json.Unmarshal(rr.Body.Bytes(), &raw)
+	if err != nil {
+		fmt.Printf("Failed to parse json response on user create: %s\n", err)
+	}
+
+	request = httptest.NewRequest("DELETE", "/user", nil)
+	request.Header.Set("Content-Type", "application/json")
+	request.Header.Set("X-Bunker-Token", rootToken)
+	//var resp http.ResponseWriter
+	rr = httptest.NewRecorder()
+	p4 := httprouter.Param{"code", userTOKEN}
+	p5 := httprouter.Param{"index", "token"}
+	ps4 := []httprouter.Param{p4, p5}
+	e.userDelete(rr, request, ps4)
+	fmt.Printf("after userDelete------------------\n%s\n\n\n", rr.Body)
+	err = json.Unmarshal(rr.Body.Bytes(), &raw)
+	if err != nil {
+		fmt.Printf("Failed to parse json response on user create: %s\n", err)
+	}
+}

src/consent_api.go

@@ -0,0 +1,132 @@
+package main
+
+import (
+	"fmt"
+	"net/http"
+	"reflect"
+
+	"github.com/julienschmidt/httprouter"
+)
+
+func (e mainEnv) consentAccept(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	var err error
+	code := ps.ByName("code")
+	index := ps.ByName("index")
+	event := audit("consent accept by "+index, code)
+	defer func() { event.submit(e.db) }()
+
+	userTOKEN := ""
+	if index == "token" {
+		if enforceUUID(w, code, event) == false {
+			return
+		}
+		userBson, _ := e.db.lookupUserRecord(code)
+		if userBson != nil {
+			userTOKEN = code
+		}
+	} else {
+		// TODO: decode url in code!
+		userBson, _ := e.db.lookupUserRecordByIndex(index, code)
+		if userBson != nil {
+			userTOKEN = userBson["token"].(string)
+		}
+	}
+	defer func() {
+		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.WriteHeader(200)
+		w.Write([]byte(`{"status":"ok"}`))
+	}()
+	records, err := getJSONPostData(r)
+	if err != nil {
+		//returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	brief := ""
+	message := ""
+	status := "accept"
+	if value, ok := records["brief"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			brief = value.(string)
+		}
+	}
+	if value, ok := records["message"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			message = value.(string)
+		}
+	}
+	if value, ok := records["status"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			status = value.(string)
+		}
+	}
+	if len(brief) == 0 {
+		//returnError(w, r, "internal error", 405, nil, event)
+		return
+	}
+	if len(message) == 0 {
+		message = brief
+	}
+	e.db.createConsentRecord(userTOKEN, index, code, brief, message, status)
+}
+
+func (e mainEnv) consentCancel(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	code := ps.ByName("code")
+	index := ps.ByName("index")
+	event := audit("consent cancel by "+index, code)
+	defer func() { event.submit(e.db) }()
+	userTOKEN := code
+	if enforceUUID(w, userTOKEN, event) == false {
+		return
+	}
+	// make sure that user is logged in here, unless he wants to cancel emails
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	records, err := getJSONPostData(r)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	brief := ""
+	if value, ok := records["brief"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			brief = value.(string)
+		}
+	}
+	if len(brief) == 0 {
+		returnError(w, r, "consent brief code is missing", 405, nil, event)
+		return
+	}
+	e.db.cancelConsentRecord(userTOKEN, brief)
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	w.Write([]byte(`{"status":"ok"}`))
+}
+
+func (e mainEnv) consentList(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	code := ps.ByName("code")
+	index := ps.ByName("index")
+	event := audit("consent list of events by "+index, code)
+	defer func() { event.submit(e.db) }()
+	userTOKEN := code
+	if enforceUUID(w, userTOKEN, event) == false {
+		return
+	}
+	// make sure that user is logged in here, unless he wants to cancel emails
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+
+	resultJSON, numRecords, err := e.db.listConsentRecords(userTOKEN)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	fmt.Printf("Total count of rows: %d\n", numRecords)
+	//fmt.Fprintf(w, "<html><head><title>title</title></head>")
+	w.Header().Set("Access-Control-Allow-Origin", "*")
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	str := fmt.Sprintf(`{"status":"ok","total":%d,"rows":%s}`, numRecords, resultJSON)
+	w.Write([]byte(str))
+}

src/consent_db.go

@@ -0,0 +1,86 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"time"
+
+	"github.com/fatih/structs"
+	"go.mongodb.org/mongo-driver/bson"
+)
+
+type consentEvent struct {
+	When    int32  `json:"when,omitempty" structs:"when"`
+	Who     string `json:"who,omitempty" structs:"who"`
+	Type    string `json:"type,omitempty" structs:"type"`
+	Token   string `json:"token,omitempty" structs:"token"`
+	Brief   string `json:"brief,omitempty" structs:"brief"`
+	Message string `json:"message,omitempty" structs:"message"`
+	Status  string `json:"status,omitempty" structs:"status"`
+}
+
+func (dbobj dbcon) createConsentRecord(userTOKEN string, usertype string, usercode string, brief string, message string, status string) {
+	now := int32(time.Now().Unix())
+	// brief can not be too long, may be hash it ?
+	if len(brief) > 64 {
+		return
+	}
+	if len(userTOKEN) > 0 {
+		// first check if this consent exists, then update
+		raw, err := dbobj.getRecord2(TblName.Consent, "token", userTOKEN, "brief", brief)
+		if err != nil {
+			fmt.Printf("error to find:%s", err)
+			return
+		}
+		if raw != nil {
+			fmt.Println("update rec")
+			// update date, status
+			bdoc := bson.M{}
+			bdoc["when"] = now
+			bdoc["status"] = status
+			dbobj.updateRecord2(TblName.Consent, "token", userTOKEN, "brief", brief, &bdoc, nil)
+			return
+		}
+	}
+	ev := consentEvent{
+		When:    now,
+		Who:     usercode,
+		Token:   userTOKEN,
+		Type:    usertype,
+		Brief:   brief,
+		Message: message,
+		Status:  status,
+	}
+	// in any case - insert record
+	fmt.Printf("insert consent record\n")
+	dbobj.createRecord(TblName.Consent, structs.Map(ev))
+}
+
+func (dbobj dbcon) cancelConsentRecord(userTOKEN string, brief string) error {
+	// brief can not be too long, may be hash it ?
+	if len(brief) > 64 {
+		return errors.New("Brief value is too long")
+	}
+	fmt.Printf("%s %s\n", userTOKEN, brief)
+	now := int32(time.Now().Unix())
+	// update date, status
+	bdoc := bson.M{}
+	bdoc["when"] = now
+	bdoc["status"] = "cancel"
+	dbobj.updateRecord2(TblName.Consent, "token", userTOKEN, "brief", brief, &bdoc, nil)
+	return nil
+}
+
+// link consent to user?
+
+func (dbobj dbcon) listConsentRecords(userTOKEN string) ([]byte, int, error) {
+	records, err := dbobj.getList(TblName.Consent, "token", userTOKEN, 0, 0)
+	if err != nil {
+		return nil, 0, err
+	}
+	count := len(records)
+	resultJSON, err := json.Marshal(records)
+	//fmt.Printf("Found multiple documents (array of pointers): %+v\n", results)
+	return resultJSON, count, nil
+}

src/cryptor.go

@@ -0,0 +1,76 @@
+package main
+
+import (
+	"crypto/aes"
+	"crypto/cipher"
+	"crypto/rand"
+	"io"
+)
+
+// shamir secret split
+// https://github.com/hashicorp/vault/tree/master/shamir
+// https://github.com/kinvolk/go-shamir
+// go get github.com/hashicorp/vault/shamir
+
+func generateRecordKey() ([]byte, error) {
+	key := make([]byte, 8)
+	if _, err := io.ReadFull(rand.Reader, key); err != nil {
+		return nil, err
+	}
+	return key, nil
+}
+
+// generate master key - 24 bytes length
+func generateMasterKey() ([]byte, error) {
+	masterKey := make([]byte, 24)
+	_, err := io.ReadFull(rand.Reader, masterKey)
+	return masterKey, err
+}
+
+func decrypt(masterKey []byte, userKey []byte, data []byte) ([]byte, error) {
+	// Load your secret key from a safe place and reuse it across multiple
+	// Seal/Open calls. (Obviously don't use this example key for anything
+	// real.) If you want to convert a passphrase to a key, use a suitable
+	// package like bcrypt or scrypt.
+	// When decoded the key should be 16 bytes (AES-128) or 32 (AES-256).
+	key := append(masterKey, userKey...)
+
+	block, err := aes.NewCipher(key)
+	if err != nil {
+		return nil, err
+	}
+
+	aesgcm, err := cipher.NewGCM(block)
+	if err != nil {
+		return nil, err
+	}
+	ciphertext := data[0 : len(data)-12]
+	nonce := data[len(data)-12:]
+	plaintext, err := aesgcm.Open(nil, nonce, ciphertext, nil)
+	return plaintext, err
+}
+
+func encrypt(masterKey []byte, userKey []byte, plaintext []byte) ([]byte, error) {
+	// We use 32 byte key (AES-256).
+	// comprising 24 master key
+	// and 8 bytes record key
+	key := append(masterKey, userKey...)
+	block, err := aes.NewCipher(key)
+	if err != nil {
+		return nil, err
+	}
+	// Never use more than 2^32 random nonces with a given key because of the risk of a repeat.
+	nonce := make([]byte, 12)
+	if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
+		return nil, err
+	}
+	aesgcm, err := cipher.NewGCM(block)
+	if err != nil {
+		return nil, err
+	}
+	ciphertext := aesgcm.Seal(nil, nonce, plaintext, nil)
+	//fmt.Printf("%x\n", ciphertext)
+	// apppend random nonce bvalue to the end
+	ciphertext = append(ciphertext, nonce...)
+	return ciphertext, nil
+}

src/email.go

@@ -0,0 +1,50 @@
+package main
+
+import (
+	"fmt"
+	"net/smtp"
+	"strings"
+)
+
+func sendCodeByEmail(code string, address string, cfg Config) {
+	/*
+		c, err := smtp.Dial(smtpServer)
+		if err != nil {
+			log.Fatal(err)
+		}
+		defer c.Close()
+		// Set the sender and recipient.
+		c.Mail("bot@paranoidguy.com")
+		c.Rcpt(address)
+		// Send the email body.
+		wc, err := c.Data()
+		if err != nil {
+			log.Fatal(err)
+			return
+		}
+		defer wc.Close()
+		buf := bytes.NewBufferString("This is the email body.")
+		if _, err = buf.WriteTo(wc); err != nil {
+			log.Fatal(err)
+			return
+		}
+		return
+	*/
+	Dest := []string{"stremovsky@gmail.com", address}
+	Subject := "Access Code"
+	bodyMessage := "Data bunker access code is " + code
+	msg := "From: " + cfg.Smtp.Sender + "\n" +
+		"To: " + strings.Join(Dest, ",") + "\n" +
+		"Subject: " + Subject + "\n" + bodyMessage
+
+	err := smtp.SendMail(cfg.Smtp.Server+":"+cfg.Smtp.Port,
+		smtp.PlainAuth("", cfg.Smtp.User, cfg.Smtp.Pass, cfg.Smtp.Server),
+		cfg.Smtp.User, Dest, []byte(msg))
+
+	if err != nil {
+		fmt.Printf("smtp error: %s", err)
+		return
+	}
+
+	fmt.Println("Mail sent successfully!")
+}

src/qldb.go

@@ -0,0 +1,747 @@
+package main
+
+// This project is using the following golang internal database:
+// https://godoc.org/modernc.org/ql
+
+// go build modernc.org/ql/ql
+// go install modernc.org/ql/ql
+
+// https://stackoverflow.com/questions/21986780/is-it-possible-to-retrieve-a-column-value-by-name-using-golang-database-sql
+
+// https://stackoverflow.com/questions/21986780/is-it-possible-to-retrieve-a-column-value-by-name-using-golang-database-sql
+
+import (
+	"crypto/md5"
+	"database/sql"
+	"fmt"
+	"log"
+	"strconv"
+
+	"go.mongodb.org/mongo-driver/bson"
+	"go.mongodb.org/mongo-driver/bson/primitive"
+	"modernc.org/ql"
+)
+
+var (
+	knownApps []string
+)
+
+type dbcon struct {
+	db        *sql.DB
+	masterKey []byte
+	hash      []byte
+}
+
+func newDB(masterKey []byte, urlurl *string) (dbcon, error) {
+	dbobj := dbcon{nil, nil, nil}
+
+	/*
+		if db, err := ql.OpenFile("./bunker.db", &ql.Options{}); err != nil {
+			fmt.Println("open db")
+			a, err := db.Info()
+			if err != nil {
+				fmt.Printf("error in db info: %s\n", err)
+			}
+			for _, v := range a.Tables {
+				fmt.Printf("dbinfo: %s\n", string(v.Name))
+			}
+			db.Close()
+		}
+	*/
+
+	ql.RegisterDriver2()
+	db, err := sql.Open("ql2", "./bql.db")
+	if err != nil {
+		log.Fatalf("Failed to open ql db: %s", err)
+	}
+	hash := md5.Sum(masterKey)
+	dbobj = dbcon{db, masterKey, hash[:]}
+	return dbobj, nil
+}
+
+func (dbobj dbcon) initDB() error {
+	var err error
+	fmt.Println("init db *****")
+	if err = initUsers(dbobj.db); err != nil {
+		return err
+	}
+	if err = initXTokens(dbobj.db); err != nil {
+		return err
+	}
+	if err = initAudit(dbobj.db); err != nil {
+		return err
+	}
+	if err = initConsent(dbobj.db); err != nil {
+		return err
+	}
+	if err = initSessions(dbobj.db); err != nil {
+		return err
+	}
+	return nil
+}
+
+func (dbobj dbcon) initUserApps() error {
+	return nil
+}
+
+func decodeFieldsValues(data interface{}) (string, string) {
+	fields := ""
+	values := ""
+	str := ""
+
+	switch t := data.(type) {
+	case primitive.M:
+		fmt.Println("format is: primitive.M")
+		for idx, val := range data.(primitive.M) {
+			if len(fields) == 0 {
+				fields = idx
+			} else {
+				fields = fields + "," + idx
+			}
+			switch t := val.(type) {
+			case string:
+				str = "\"" + val.(string) + "\""
+			case int:
+				str = strconv.Itoa(val.(int))
+			case int32:
+				str = strconv.FormatInt(int64(val.(int32)), 10)
+			default:
+				fmt.Printf("wrong type: %s\n", t)
+			}
+			if len(values) == 0 {
+				values = str
+			} else {
+				values = values + "," + str
+			}
+		}
+	case *primitive.M:
+		fmt.Println("format is: *primitive.M")
+		for idx, val := range *data.(*primitive.M) {
+			if len(fields) == 0 {
+				fields = idx
+			} else {
+				fields = fields + "," + idx
+			}
+			switch t := val.(type) {
+			case string:
+				str = "\"" + val.(string) + "\""
+			case int:
+				str = strconv.Itoa(val.(int))
+			case int32:
+				str = strconv.FormatInt(int64(val.(int32)), 10)
+			default:
+				fmt.Printf("wrong type: %s\n", t)
+			}
+			if len(values) == 0 {
+				values = str
+			} else {
+				values = values + "," + str
+			}
+		}
+	case map[string]interface{}:
+		fmt.Println("format is: map[string]interface{}")
+		for idx, val := range data.(map[string]interface{}) {
+			if len(fields) == 0 {
+				fields = idx
+			} else {
+				fields = fields + "," + idx
+			}
+			switch t := val.(type) {
+			case string:
+				str = "\"" + val.(string) + "\""
+			case int:
+				str = strconv.Itoa(val.(int))
+			case int32:
+				str = strconv.FormatInt(int64(val.(int32)), 10)
+			default:
+				fmt.Printf("wrong type: %s\n", t)
+			}
+			if len(values) == 0 {
+				values = str
+			} else {
+				values = values + "," + str
+			}
+		}
+	default:
+		fmt.Printf("XXXXXX wrong type: %T\n", t)
+	}
+	return fields, values
+}
+
+func decodeForCleanup(data interface{}) string {
+	fields := ""
+
+	switch t := data.(type) {
+	case primitive.M:
+		for idx, _ := range data.(primitive.M) {
+			if len(fields) == 0 {
+				fields = idx + "=null"
+			} else {
+				fields = fields + "," + idx + "=null"
+			}
+		}
+		return fields
+	case map[string]interface{}:
+		for idx, _ := range data.(map[string]interface{}) {
+			if len(fields) == 0 {
+				fields = idx + "=null"
+			} else {
+				fields = fields + "," + idx + "=null"
+			}
+		}
+	default:
+		fmt.Printf("decodeForCleanup: wrong type: %s\n", t)
+	}
+
+	return fields
+}
+
+func decodeForUpdate(bdoc *bson.M, bdel *bson.M) string {
+	fields := ""
+	str := ""
+
+	if bdoc != nil {
+		/*
+			switch t := *bdoc.(type) {
+			default:
+				fmt.Printf("Type is %T\n", t)
+			}
+		*/
+
+		for idx, val := range *bdoc {
+			switch t := val.(type) {
+			case string:
+				str = "\"" + val.(string) + "\""
+			case int:
+				str = strconv.Itoa(val.(int))
+			case int32:
+				str = strconv.FormatInt(int64(val.(int32)), 10)
+			default:
+				fmt.Printf("wrong type: %s\n", t)
+			}
+			if len(fields) == 0 {
+				fields = idx + "=" + str
+			} else {
+				fields = fields + "," + idx + "=" + str
+			}
+		}
+	}
+	if bdel != nil {
+		for idx, _ := range *bdel {
+			if len(fields) == 0 {
+				fields = idx + "=null"
+			} else {
+				fields = fields + "," + idx + "=null"
+			}
+		}
+	}
+	return fields
+}
+
+func getTable(t Tbl) string {
+	switch t {
+	case TblName.Users:
+		return "users"
+	case TblName.Audit:
+		return "audit"
+	case TblName.Consent:
+		return "consent"
+	case TblName.Xtokens:
+		return "xtokens"
+	case TblName.Sessions:
+		return "sessions"
+	}
+	return "users"
+}
+
+func (dbobj dbcon) createRecordInTable(tbl string, data interface{}) (int, error) {
+	fields, values := decodeFieldsValues(data)
+	q := "insert into " + tbl + " (" + fields + ") values (" + values + ");"
+	fmt.Printf("q: %s\n", q)
+
+	tx, err := dbobj.db.Begin()
+	if err != nil {
+		return 0, err
+	}
+	_, err = tx.Exec(q)
+	if err != nil {
+		return 0, err
+	}
+	if err = tx.Commit(); err != nil {
+		return 0, err
+	}
+	return 1, nil
+}
+
+func (dbobj dbcon) createRecord(t Tbl, data interface{}) (int, error) {
+	//if reflect.TypeOf(value) == reflect.TypeOf("string")
+	tbl := getTable(t)
+	return dbobj.createRecordInTable(tbl, data)
+}
+
+func (dbobj dbcon) countRecords(t Tbl, keyName string, keyValue string) (int64, error) {
+	tbl := getTable(t)
+	q := "select count(*) from " + tbl + " WHERE " + keyName + "'" + keyValue + "';"
+	fmt.Printf("q: %s\n", q)
+
+	tx, err := dbobj.db.Begin()
+	if err != nil {
+		return 0, err
+	}
+	row := tx.QueryRow(q)
+	// Columns
+	var count int
+	err = row.Scan(&count)
+	if err != nil {
+		return 0, err
+	}
+	if err = tx.Commit(); err != nil {
+		return 0, err
+	}
+	return int64(count), nil
+}
+
+func (dbobj dbcon) updateRecord(t Tbl, keyName string, keyValue string, bdoc *bson.M) (int64, error) {
+	table := getTable(t)
+	filter := keyName + "=\"" + keyValue + "\""
+	return dbobj.updateRecordInTableDo(table, filter, bdoc, nil)
+}
+
+func (dbobj dbcon) updateRecordInTable(table string, keyName string, keyValue string, bdoc *bson.M) (int64, error) {
+	filter := keyName + "=\"" + keyValue + "\""
+	return dbobj.updateRecordInTableDo(table, filter, bdoc, nil)
+}
+
+func (dbobj dbcon) updateRecord2(t Tbl, keyName string, keyValue string,
+	keyName2 string, keyValue2 string, bdoc *bson.M, bdel *bson.M) (int64, error) {
+	table := getTable(t)
+	filter := keyName + "=\"" + keyValue + "\" AND " + keyName2 + "=\"" + keyValue2 + "\""
+	return dbobj.updateRecordInTableDo(table, filter, bdoc, bdel)
+}
+
+func (dbobj dbcon) updateRecordInTable2(table string, keyName string,
+	keyValue string, keyName2 string, keyValue2 string, bdoc *bson.M, bdel *bson.M) (int64, error) {
+	filter := keyName + "=\"" + keyValue + "\" AND " + keyName2 + "=\"" + keyValue2 + "\""
+	return dbobj.updateRecordInTableDo(table, filter, bdoc, bdel)
+}
+
+func (dbobj dbcon) updateRecordInTableDo(table string, filter string, bdoc *bson.M, bdel *bson.M) (int64, error) {
+	op := decodeForUpdate(bdoc, bdel)
+	q := "update " + table + " SET " + op + " WHERE " + filter
+	fmt.Printf("q: %s\n", q)
+
+	tx, err := dbobj.db.Begin()
+	if err != nil {
+		return 0, err
+	}
+	defer tx.Rollback()
+	result, err := tx.Exec(q)
+	if err != nil {
+		return 0, err
+	}
+	if err = tx.Commit(); err != nil {
+		return 0, err
+	}
+	num, err := result.RowsAffected()
+	return num, err
+}
+
+func (dbobj dbcon) getRecord(t Tbl, keyName string, keyValue string) (bson.M, error) {
+	tbl := getTable(t)
+	return dbobj.getRecordInTable(tbl, keyName, keyValue)
+}
+
+func (dbobj dbcon) getRecordInTable(table string, keyName string, keyValue string) (bson.M, error) {
+	q := "select * from " + table + " WHERE " + keyName + "=\"" + keyValue + "\""
+	return dbobj.getRecordInTableDo(q)
+}
+
+func (dbobj dbcon) getRecord2(t Tbl, keyName string, keyValue string,
+	keyName2 string, keyValue2 string) (bson.M, error) {
+	tbl := getTable(t)
+	return dbobj.getRecordInTable2(tbl, keyName, keyValue, keyName2, keyValue2)
+}
+
+func (dbobj dbcon) getRecordInTable2(table string, keyName string, keyValue string,
+	keyName2 string, keyValue2 string) (bson.M, error) {
+	q := "select * from " + table + " WHERE " + keyName + "=\"" + keyValue + "\" AND " +
+		keyName2 + "=\"" + keyValue2 + "\""
+	return dbobj.getRecordInTableDo(q)
+}
+
+func (dbobj dbcon) getRecordInTableDo(q string) (bson.M, error) {
+	fmt.Printf("q: %s\n", q)
+	tx, err := dbobj.db.Begin()
+	if err != nil {
+		return nil, err
+	}
+	defer tx.Rollback()
+	rows, err := tx.Query(q)
+	if err == sql.ErrNoRows {
+		fmt.Println("nothing found")
+		return nil, nil
+	} else if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	columnNames, err := rows.Columns()
+	if err != nil {
+		return nil, err
+	}
+	//fmt.Printf("names: %s\n", columnNames)
+	if err := rows.Err(); err != nil {
+		log.Fatal(err)
+	}
+	//pointers := make([]interface{}, len(columnNames))
+	recBson := bson.M{}
+	rows.Next()
+	//for rows.Next() {
+	//fmt.Println("parsing result line")
+	columnPointers := make([]interface{}, len(columnNames))
+	//for i, _ := range columnNames {
+	//		columnPointers[i] = new(interface{})
+	//}
+	columns := make([]interface{}, len(columnNames))
+	for i, _ := range columns {
+		columnPointers[i] = &columns[i]
+	}
+
+	err = rows.Scan(columnPointers...)
+	if err == sql.ErrNoRows {
+		fmt.Println("nothing found")
+		return nil, nil
+	}
+	if err != nil {
+		fmt.Printf("nothing found: %s\n", err)
+		return nil, nil
+	}
+	for i, colName := range columnNames {
+		switch t := columns[i].(type) {
+		case string:
+			recBson[colName] = columns[i]
+		case []uint8:
+			recBson[colName] = string(columns[i].([]uint8))
+		case int64:
+			recBson[colName] = int32(columns[i].(int64))
+		case nil:
+			//fmt.Printf("is nil, not interesting\n")
+		default:
+			fmt.Printf("field: %s - %s, unknown: %s - %T\n", colName, columns[i], t, t)
+		}
+	}
+	//}
+	err = rows.Close()
+	if err == sql.ErrNoRows {
+		fmt.Println("nothing found2")
+		return nil, nil
+	} else if err != nil {
+		return nil, err
+	}
+	if len(recBson) == 0 {
+		fmt.Println("no result!!!")
+		return nil, nil
+	}
+	if err = tx.Commit(); err != nil {
+		return recBson, err
+	}
+	return recBson, nil
+}
+
+func (dbobj dbcon) deleteRecord(t Tbl, keyName string, keyValue string) (int64, error) {
+	tbl := getTable(t)
+	return dbobj.deleteRecordInTable(tbl, keyName, keyValue)
+}
+
+func (dbobj dbcon) deleteRecordInTable(table string, keyName string, keyValue string) (int64, error) {
+	q := "delete from " + table + " WHERE " + keyName + "=\"" + keyValue + "\""
+	fmt.Printf("q: %s\n", q)
+
+	tx, err := dbobj.db.Begin()
+	if err != nil {
+		return 0, err
+	}
+	defer tx.Rollback()
+	result, err := tx.Exec(q)
+	if err != nil {
+		return 0, err
+	}
+	if err = tx.Commit(); err != nil {
+		return 0, err
+	}
+	num, err := result.RowsAffected()
+	return num, err
+}
+
+func (dbobj dbcon) cleanupRecord(t Tbl, keyName string, keyValue string, data interface{}) (int64, error) {
+	tbl := getTable(t)
+	cleanup := decodeForCleanup(data)
+	q := "update " + tbl + " SET " + cleanup + " WHERE " + keyName + "=\"" + keyValue + "\""
+	fmt.Printf("q: %s\n", q)
+
+	tx, err := dbobj.db.Begin()
+	if err != nil {
+		return 0, err
+	}
+	defer tx.Rollback()
+	result, err := tx.Exec(q)
+	if err != nil {
+		return 0, err
+	}
+	if err = tx.Commit(); err != nil {
+		return 0, err
+	}
+	num, err := result.RowsAffected()
+	return num, err
+}
+
+func (dbobj dbcon) getList(t Tbl, keyName string, keyValue string, start int32, limit int32) ([]bson.M, error) {
+	fmt.Println("TODO")
+	return nil, nil
+}
+
+func (dbobj dbcon) getAllTables() ([]string, error) {
+	//for nm, tab := range dbobj.db.root.tables {
+	//
+	//	}
+	// HasNextResultSet()
+	a := []string{"aaa"}
+	a = append(a, "test123")
+	return a, nil
+}
+func (dbobj dbcon) indexNewApp(appName string) {
+	if contains(knownApps, appName) == false {
+		// it is a new app, create an index
+		fmt.Printf("This is a new app, creating index for :%s\n", appName)
+
+		tx, err := dbobj.db.Begin()
+		if err != nil {
+			return
+		}
+		defer tx.Rollback()
+		_, err = tx.Exec("CREATE TABLE IF NOT EXISTS " + appName + ` (
+	  		token STRING,
+	  		md5 STRING,
+	  		data STRING,
+	  		status STRING,
+	  		when int
+		);`)
+		if err != nil {
+			return
+		}
+		_, err = tx.Exec("CREATE INDEX IF NOT EXISTS " + appName + "_token ON " + appName + " (token)")
+		if err != nil {
+			return
+		}
+		if err = tx.Commit(); err != nil {
+			return
+		}
+		knownApps = append(knownApps, appName)
+	}
+	return
+}
+
+/*
+BEGIN TRANSACTION;
+	CREATE TABLE Orders (CustomerID int, Date time);
+	CREATE INDEX OrdersID ON Orders (id());
+	CREATE INDEX OrdersDate ON Orders (Date);
+	CREATE TABLE Items (OrderID int, ProductID int, Qty int);
+	CREATE INDEX ItemsOrderID ON Items (OrderID);
+COMMIT;
+*/
+
+func initUsers(db *sql.DB) error {
+	tx, err := db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	_, err = tx.Exec(`
+	CREATE TABLE IF NOT EXISTS users (
+	  token STRING,
+	  key STRING,
+	  md5 STRING,
+	  loginidx STRING,
+	  emailidx STRING,
+	  phoneidx STRING,
+	  tempcode STRING,
+	  tempcodeexp int,
+	  data string
+	);
+	`)
+	if err != nil {
+		return err
+	}
+	fmt.Println("going to create indexes")
+	_, err = tx.Exec(`CREATE INDEX users_token ON users (token);`)
+	if err != nil {
+		fmt.Println("error in create index")
+		return err
+	}
+	_, err = tx.Exec(`CREATE INDEX users_login ON users (loginidx);`)
+	if err != nil {
+		return err
+	}
+	_, err = tx.Exec(`CREATE INDEX users_email ON users (emailidx);`)
+	if err != nil {
+		return err
+	}
+	_, err = tx.Exec(`CREATE INDEX users_phone ON users (phoneidx);`)
+	if err != nil {
+		return err
+	}
+	if err = tx.Commit(); err != nil {
+		return err
+	}
+	return nil
+}
+
+func initXTokens(db *sql.DB) error {
+	tx, err := db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	_, err = tx.Exec(`
+	CREATE TABLE IF NOT EXISTS xtokens (
+	  xtoken STRING,
+	  token STRING,
+	  type STRING,
+	  app STRING,
+	  fields STRING,
+	  endtime int32
+	);
+	`)
+	if err != nil {
+		return err
+	}
+	_, err = tx.Exec(`CREATE INDEX xtokens_xtoken ON xtokens (xtoken);`)
+	if err != nil {
+		return err
+	}
+	_, err = tx.Exec(`CREATE INDEX xtokens_type ON xtokens (type);`)
+	if err != nil {
+		return err
+	}
+	if err = tx.Commit(); err != nil {
+		return err
+	}
+	return nil
+}
+
+/*
+	When   int32  `json:"when"`
+	Who    string `json:"who"`
+	Record string `json:"record"`
+	App    string `json:"app"`
+	Title  string `json:"title"`
+	Status string `json:"status"`
+	Msg    string `json:"msg"`
+	Debug  string `json:"debug"`
+	Before string `json:"before"`
+	After  string `json:"after"`
+	Meta   string `json:"meta"`
+*/
+
+func initAudit(db *sql.DB) error {
+	tx, err := db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	_, err = tx.Exec(`
+	CREATE TABLE IF NOT EXISTS audit (
+	  record STRING,
+	  who STRING,
+	  app STRING,
+	  title STRING,
+	  status STRING,
+	  msg STRING,
+	  debug STRING,
+	  before STRING,
+	  after STRING,
+	  meta STRING,
+	  when int
+	);
+	`)
+	if err != nil {
+		return err
+	}
+	_, err = tx.Exec(`CREATE INDEX audit_record ON audit (record);`)
+	if err != nil {
+		return err
+	}
+	if err = tx.Commit(); err != nil {
+		return err
+	}
+	return nil
+}
+
+/*
+	When    int32  `json:"when,omitempty"`
+	Who     string `json:"who,omitempty"`
+	Type    string `json:"type,omitempty"`
+	Token   string `json:"token,omitempty"`
+	Brief   string `json:"brief,omitempty"`
+	Message string `json:"message,omitempty"`
+	Status  string `json:"status,omitempty"`
+*/
+
+func initConsent(db *sql.DB) error {
+	tx, err := db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	_, err = tx.Exec(`
+	CREATE TABLE IF NOT EXISTS consent (
+	  who STRING,
+	  type STRING,
+	  token STRING,
+	  brief STRING,
+	  message STRING,
+	  status STRING,
+	  when int
+	);
+	`)
+	if err != nil {
+		return err
+	}
+	_, err = tx.Exec(`CREATE INDEX consent_token ON consent (token);`)
+	if err != nil {
+		return err
+	}
+	if err = tx.Commit(); err != nil {
+		return err
+	}
+	return nil
+}
+
+func initSessions(db *sql.DB) error {
+	tx, err := db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	_, err = tx.Exec(`
+	CREATE TABLE IF NOT EXISTS sessions (
+	  token STRING,
+	  session STRING,
+	  meta STRING,
+	  when int,
+	  endtime int
+	);
+	`)
+	if err != nil {
+		return err
+	}
+	_, err = tx.Exec(`CREATE INDEX sessions_token ON sessions (token);`)
+	if err != nil {
+		return err
+	}
+	if err = tx.Commit(); err != nil {
+		return err
+	}
+	return nil
+}

src/sessions_api.go

@@ -0,0 +1,13 @@
+package main
+
+import (
+	"net/http"
+
+	"github.com/julienschmidt/httprouter"
+)
+
+func (e mainEnv) newSession(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	uuidCode := ps.ByName("uuidcode")
+	event := audit("create new session", uuidCode)
+	defer func() { event.submit(e.db) }()
+}

src/sessions_db.go

@@ -0,0 +1,103 @@
+package main
+
+import (
+	"crypto/sha256"
+	"encoding/base64"
+	"errors"
+	"time"
+
+	uuid "github.com/hashicorp/go-uuid"
+	"go.mongodb.org/mongo-driver/bson"
+)
+
+type sessionEvent struct {
+	When int32
+	Meta []byte
+}
+
+func (dbobj dbcon) generateUserSession(userTOKEN string, clientip string, expiration string, meta []byte) (string, error) {
+	if len(expiration) == 0 {
+		return "", errors.New("failed to parse expiration")
+	}
+	endtime, err := parseExpiration(expiration)
+	if err != nil {
+		return "", err
+	}
+	encodedStr, err := dbobj.userEncrypt(userTOKEN, meta)
+	if err != nil {
+		return "", err
+	}
+	tokenUUID, err := uuid.GenerateUUID()
+	if err != nil {
+		return "", err
+	}
+	bdoc := bson.M{}
+	bdoc["token"] = userTOKEN
+	bdoc["session"] = tokenUUID
+	bdoc["endtime"] = endtime
+	bdoc["meta"] = encodedStr
+	if len(clientip) > 0 {
+		idxString := append(dbobj.hash, []byte(clientip)...)
+		idxStringHash := sha256.Sum256(idxString)
+		bdoc["clientipidx"] = base64.StdEncoding.EncodeToString(idxStringHash[:])
+	}
+	_, err = dbobj.createRecord(TblName.Sessions, bdoc)
+	if err != nil {
+		return "", err
+	}
+	return tokenUUID, nil
+}
+
+func (dbobj dbcon) getUserSession(sessionUUID string) ([]byte, error) {
+	record, err := dbobj.getRecord(TblName.Sessions, "session", sessionUUID)
+	if record == nil || err != nil {
+		return nil, errors.New("failed to authenticate")
+	}
+	// check expiration
+	now := int32(time.Now().Unix())
+	if now > record["endtime"].(int32) {
+		return nil, errors.New("session expired")
+	}
+	userTOKEN := record["token"].(string)
+	encData0 := record["meta"].(string)
+	decrypted, err := dbobj.userDecrypt(userTOKEN, encData0)
+	if err != nil {
+		return nil, err
+	}
+	return decrypted, err
+}
+
+func (dbobj dbcon) getUserSessionByToken(userTOKEN string) ([]*sessionEvent, int64, error) {
+
+	userBson, err := dbobj.lookupUserRecord(userTOKEN)
+	if userBson == nil || err != nil {
+		// not found
+		return nil, 0, err
+	}
+	userKey := userBson["key"].(string)
+	recordKey, err := base64.StdEncoding.DecodeString(userKey)
+	if err != nil {
+		return nil, 0, err
+	}
+
+	count, err := dbobj.countRecords(TblName.Sessions, "token", userTOKEN)
+	if err != nil {
+		return nil, 0, err
+	}
+
+	records, err := dbobj.getList(TblName.Sessions, "token", userTOKEN, 0, 0)
+	if err != nil {
+		return nil, 0, err
+	}
+
+	var results []*sessionEvent
+	for _, element := range records {
+		encData0 := element["meta"].(string)
+		encData, _ := base64.StdEncoding.DecodeString(encData0)
+		decrypted, _ := decrypt(dbobj.masterKey, recordKey, encData)
+		sEvent := sessionEvent{0, decrypted}
+		results = append(results, &sEvent)
+	}
+
+	return results, count, err
+}

src/sms.go

@@ -0,0 +1,35 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/url"
+	"strings"
+)
+
+func sendCodeByPhone(code string, address string, cfg Config) {
+	urlStr := "https://api.twilio.com/2010-04-01/Accounts/" + cfg.Sms.Twilio_account + "/Messages.json"
+	fmt.Printf("url %s\n", urlStr)
+	msgData := url.Values{}
+	msgData.Set("To", address)
+	msgData.Set("From", cfg.Sms.Twilio_from)
+	msgData.Set("Body", "Data Bunker code "+code)
+	msgDataReader := *strings.NewReader(msgData.Encode())
+	client := &http.Client{}
+	req, _ := http.NewRequest("POST", urlStr, &msgDataReader)
+	req.SetBasicAuth(cfg.Sms.Twilio_account, cfg.Sms.Twilio_token)
+	req.Header.Add("Accept", "application/json")
+	req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
+	resp, _ := client.Do(req)
+	if resp.StatusCode >= 200 && resp.StatusCode < 300 {
+		var data map[string]interface{}
+		decoder := json.NewDecoder(resp.Body)
+		err := decoder.Decode(&data)
+		if err == nil {
+			fmt.Println(data["sid"])
+		}
+	} else {
+		fmt.Println(resp.Status)
+	}
+}

src/tokens_test.go

@@ -0,0 +1,133 @@
+package main
+
+import (
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	uuid "github.com/hashicorp/go-uuid"
+)
+
+func Test_UserTempToken(t *testing.T) {
+	masterKey, err := hex.DecodeString("71c65924336c5e6f41129b6f0540ad03d2a8bf7e9b10db72")
+	db, _ := newDB(masterKey, nil)
+
+	var parsedData userJSON
+	parsedData.jsonData = []byte(`{"login":"start","key1":"bbb","key2":[10,20]}`)
+	parsedData.loginIdx = "start"
+	userTOKEN, err := db.createUserRecord(parsedData, nil)
+	if err != nil {
+		t.Fatalf("Failed to create user: %s", err)
+	}
+	fmt.Printf("user token generated: %s\n", userTOKEN)
+	fields := "key1,key2.1"
+	expiration := "7d"
+	userToken, err := db.generateUserTempXToken(userTOKEN, fields, expiration, "")
+	if err != nil {
+		t.Fatalf("Failed to generate user token: %s ", err)
+	}
+	if userToken == "" {
+		t.Fatalf("Failed to generate user token")
+	}
+	_, err = db.deleteUserRecord(userTOKEN)
+	if err != nil {
+		t.Fatalf("Failed to delete user: %s", err)
+	}
+}
+
+func Test_UserTempToken2(t *testing.T) {
+	masterKey, err := hex.DecodeString("71c65924336c5e6f41129b6f0540ad03d2a8bf7e9b10db72")
+	db, _ := newDB(masterKey, nil)
+
+	userTOKEN, err := uuid.GenerateUUID()
+	fields := "abc"
+	expiration := "7d"
+	_, err = db.generateUserTempXToken(userTOKEN, fields, expiration, "")
+	if err == nil {
+		t.Fatalf("Should failed to generate user token")
+	}
+}
+
+func helpCreateUserXToken(uuidCode string, tokenJSON string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("POST",
+		"http://localhost:3000/v1/xtoken/"+uuidCode,
+		strings.NewReader(tokenJSON))
+	rr := httptest.NewRecorder()
+	request.Header.Set("Content-Type", "application/json")
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func TestAPIToken(t *testing.T) {
+	jsonData := `{"email":"stremovsky@gmail.com","phone":"0524486622","fname":"Yuli","lname":"Stremovsky","tz":"323xxxxx","password":"123456","address":"Y-d habanim 7","city":"Petah-Tiqva","btest":true,"numtest":123,"testnul":null}`
+	raw, err := helpCreateUser(jsonData)
+	if err != nil {
+		if strings.Contains(err.Error(), "duplicate") {
+			raw, err = helpGetUser("email", "stremovsky@gmail.com")
+		} else {
+			t.Fatalf("error: %s", err)
+		}
+	}
+	status := raw["status"].(string)
+	if status == "error" {
+		if strings.Contains(raw["message"].(string), "duplicate") {
+			raw, err = helpGetUser("email", "stremovsky@gmail.com")
+		} else {
+			t.Fatalf("Failed to create user: %s", raw["message"].(string))
+		}
+	}
+	userTOKEN := raw["token"].(string)
+	fields := "phone,field1,field2"
+	tokenJSON := fmt.Sprintf(`{"fields":"%s","expiration":"1d"}`, fields)
+	raw2, err := helpCreateUserXToken(userTOKEN, tokenJSON)
+	if err != nil {
+		t.Fatalf("error: %s", err)
+	}
+	token := raw2["xtoken"].(string)
+	fmt.Printf("**** Result token : %s\n", token)
+	raw3, err := helpGetUserAppList(userTOKEN)
+	fmt.Printf("apps: %s\n", raw3["apps"])
+	helpCreateUserApp(userTOKEN, "qq", `{"custom":1}`)
+	raw3, err = helpGetUserAppList(userTOKEN)
+	fmt.Printf("apps: %s\n", raw3["apps"])
+}
+
+func Test_UserAppToken(t *testing.T) {
+	masterKey, err := hex.DecodeString("71c65924336c5e6f41129b6f0540ad03d2a8bf7e9b10db72")
+	db, _ := newDB(masterKey, nil)
+
+	var parsedData userJSON
+	parsedData.jsonData = []byte(`{"login":"start","field":"bbb"}`)
+	parsedData.loginIdx = "start"
+	userTOKEN, err := db.createUserRecord(parsedData, nil)
+	fields := "abc"
+	expiration := "7d"
+	appName := "test"
+	userXToken, err := db.generateUserTempXToken(userTOKEN, fields, expiration, appName)
+	if err != nil {
+		t.Fatalf("Failed to generate user token: %s ", err)
+	}
+	if userXToken == "" {
+		t.Fatalf("Failed to generate user token")
+	}
+	appName = "test2"
+	userXToken, err = db.generateUserTempXToken(userTOKEN, fields, expiration, appName)
+	if err == nil {
+		t.Fatalf("Using unknown app, should fail.")
+	}
+	if userXToken != "" {
+		t.Fatalf("Should fail to generate user token")
+	}
+	_, err = db.deleteUserRecord(userTOKEN)
+	if err != nil {
+		t.Fatalf("Failed to delete user: %s", err)
+	}
+}

src/userapps_api.go

@@ -0,0 +1,149 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+
+	"github.com/julienschmidt/httprouter"
+)
+
+func (e mainEnv) userappNew(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	userTOKEN := ps.ByName("token")
+	appName := ps.ByName("appname")
+	event := auditApp("create user app record", userTOKEN, appName)
+	defer func() { event.submit(e.db) }()
+
+	if enforceUUID(w, userTOKEN, event) == false {
+		return
+	}
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	if isValidApp(appName) == false {
+		returnError(w, r, "bad appname", 405, nil, event)
+		return
+	}
+
+	data, err := getJSONPostData(r)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	jsonData, err := json.Marshal(data)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	_, err = e.db.createAppRecord(jsonData, userTOKEN, appName, event)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	returnUUID(w, userTOKEN)
+	return
+}
+
+func (e mainEnv) userappChange(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	userTOKEN := ps.ByName("token")
+	appName := ps.ByName("appname")
+	event := auditApp("change user app record", userTOKEN, appName)
+	defer func() { event.submit(e.db) }()
+
+	if enforceUUID(w, userTOKEN, event) == false {
+		return
+	}
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	if isValidApp(appName) == false {
+		returnError(w, r, "bad appname", 405, nil, event)
+		return
+	}
+
+	data, err := getJSONPostData(r)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	jsonData, err := json.Marshal(data)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	_, err = e.db.updateAppRecord(jsonData, userTOKEN, appName, event)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	returnUUID(w, userTOKEN)
+	return
+}
+
+func (e mainEnv) userappList(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	userTOKEN := ps.ByName("token")
+	event := audit("get user app list", userTOKEN)
+	defer func() { event.submit(e.db) }()
+
+	if enforceUUID(w, userTOKEN, event) == false {
+		return
+	}
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	result, err := e.db.listUserApps(userTOKEN)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	fmt.Fprintf(w, `{"status":"ok","token":"%s","apps":%s}`, userTOKEN, result)
+}
+
+func (e mainEnv) userappGet(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	userTOKEN := ps.ByName("token")
+	appName := ps.ByName("appname")
+	event := auditApp("get user app record", userTOKEN, appName)
+	defer func() { event.submit(e.db) }()
+
+	if enforceUUID(w, userTOKEN, event) == false {
+		return
+	}
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	if isValidApp(appName) == false {
+		returnError(w, r, "bad appname", 405, nil, event)
+		return
+	}
+
+	resultJSON, err := e.db.getUserApp(userTOKEN, appName)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	if resultJSON == nil {
+		returnError(w, r, "not found", 405, nil, event)
+		return
+	}
+	finalJSON := fmt.Sprintf(`{"status":"ok","token":"%s","data":%s}`, userTOKEN, resultJSON)
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	w.Write([]byte(finalJSON))
+}
+
+func (e mainEnv) appList(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	fmt.Printf("/APPLIST\n")
+	if e.enforceAuth(w, r, nil) == false {
+		return
+	}
+	result, err := e.db.listAllApps()
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, nil)
+		return
+	}
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	fmt.Fprintf(w, `{"status":"ok","apps":%s}`, result)
+}

src/userapps_db.go

@@ -0,0 +1,175 @@
+package main
+
+import (
+	"crypto/md5"
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+
+	jsonpatch "github.com/evanphx/json-patch"
+	"go.mongodb.org/mongo-driver/bson"
+)
+
+func (dbobj dbcon) getUserApp(userTOKEN string, appName string) ([]byte, error) {
+
+	record, err := dbobj.getRecordInTable("app_"+appName, "token", userTOKEN)
+	if err != nil {
+		return nil, err
+	}
+	if record == nil {
+		return nil, nil
+	}
+	encData0 := record["data"].(string)
+	return dbobj.userDecrypt(userTOKEN, encData0)
+}
+
+func (dbobj dbcon) createAppRecord(jsonData []byte, userTOKEN string, appName string, event *auditEvent) (string, error) {
+	fmt.Printf("createAppRecord app is : %s\n", appName)
+	encodedStr, err := dbobj.userEncrypt(userTOKEN, jsonData)
+	if err != nil {
+		return userTOKEN, err
+	}
+	dbobj.indexNewApp("app_" + appName)
+
+	//var bdoc interface{}
+	bdoc := bson.M{}
+	bdoc["data"] = encodedStr
+	//it is ok to use md5 here, it is only for data sanity
+	md5Hash := md5.Sum([]byte(encodedStr))
+	bdoc["md5"] = base64.StdEncoding.EncodeToString(md5Hash[:])
+	bdoc["token"] = userTOKEN
+	if event != nil {
+		event.After = encodedStr
+		event.App = appName
+		event.Record = userTOKEN
+	}
+	//fmt.Println("creating new app")
+	record, err := dbobj.getRecordInTable("app_"+appName, "token", userTOKEN)
+	if record != nil {
+		fmt.Println("update user app")
+		_, err = dbobj.updateRecordInTable("app_"+appName, "token", userTOKEN, &bdoc)
+	} else {
+		_, err = dbobj.createRecordInTable("app_"+appName, bdoc)
+	}
+	return userTOKEN, err
+}
+
+func (dbobj dbcon) updateAppRecord(jsonDataPatch []byte, userTOKEN string, appName string, event *auditEvent) (string, error) {
+	//_, err = collection.InsertOne(context.TODO(), bson.M{"name": "The Go Language2", "genre": "Coding", "authorId": "4"})
+	userBson, err := dbobj.lookupUserRecord(userTOKEN)
+	if userBson == nil || err != nil {
+		// not found
+		return userTOKEN, err
+	}
+	// get user key
+	userKey := userBson["key"].(string)
+	recordKey, err := base64.StdEncoding.DecodeString(userKey)
+	if err != nil {
+		return userTOKEN, err
+	}
+
+	record, err := dbobj.getRecordInTable("app_"+appName, "token", userTOKEN)
+	if err != nil {
+		return userTOKEN, err
+	}
+	if record == nil {
+		return userTOKEN, errors.New("user app record not found")
+	}
+	sig := record["md5"].(string)
+	encData0 := record["data"].(string)
+	encData, err := base64.StdEncoding.DecodeString(encData0)
+	decrypted, err := decrypt(dbobj.masterKey, recordKey, encData)
+
+	// merge
+	fmt.Printf("old json: %s\n", decrypted)
+	fmt.Printf("json patch: %s\n", jsonDataPatch)
+	newJSON, err := jsonpatch.MergePatch(decrypted, jsonDataPatch)
+	fmt.Printf("result: %s\n", newJSON)
+	bdoc := bson.M{}
+	encoded, err := encrypt(dbobj.masterKey, recordKey, newJSON)
+	encodedStr := base64.StdEncoding.EncodeToString(encoded)
+	bdoc["data"] = encodedStr
+	//it is ok to use md5 here, it is only for data sanity
+	md5Hash := md5.Sum([]byte(encodedStr))
+	bdoc["md5"] = base64.StdEncoding.EncodeToString(md5Hash[:])
+	bdoc["token"] = userTOKEN
+
+	// here I add md5 of the original record to filter
+	// to make sure this record was not change by other thread
+	fmt.Println("update user app")
+	result, err := dbobj.updateRecordInTable2("app_"+appName, "token", userTOKEN, "md5", sig, &bdoc, nil)
+	if err != nil {
+		return userTOKEN, err
+	}
+	if event != nil {
+		event.Before = encData0
+		event.After = encodedStr
+		if result > 0 {
+			event.Status = "ok"
+		} else {
+			event.Status = "failed"
+			event.Msg = "failed to update"
+		}
+	}
+	return userTOKEN, nil
+}
+
+// go over app collections and check if we have user record inside
+func (dbobj dbcon) listUserApps(userTOKEN string) ([]byte, error) {
+	//_, err = collection.InsertOne(context.TODO(), bson.M{"name": "The Go Language2", "genre": "Coding", "authorId": "4"})
+	record, err := dbobj.lookupUserRecord(userTOKEN)
+	if record == nil || err != nil {
+		// not found
+		return nil, err
+	}
+	allCollections, err := dbobj.getAllTables()
+	var result []string
+	for _, colName := range allCollections {
+		if strings.HasPrefix(colName, "app_") {
+			record, err := dbobj.getRecordInTable(colName, "token", userTOKEN)
+			if err != nil {
+				return nil, err
+			}
+			if record != nil {
+				result = append(result, colName[4:])
+			}
+		}
+	}
+	fmt.Printf("returning: %s\n", result)
+	resultJSON, err := json.Marshal(result)
+	return resultJSON, err
+}
+
+func (dbobj dbcon) listAllAppsOnly() ([]string, error) {
+	//fmt.Println("dump list of collections")
+	allCollections, err := dbobj.getAllTables()
+	if err != nil {
+		return nil, err
+	}
+	var result []string
+	for _, colName := range allCollections {
+		if strings.HasPrefix(colName, "app_") {
+			result = append(result, colName[4:])
+		}
+	}
+	return result, nil
+}
+
+func (dbobj dbcon) listAllApps() ([]byte, error) {
+	//fmt.Println("dump list of collections")
+	allCollections, err := dbobj.getAllTables()
+	if err != nil {
+		return nil, err
+	}
+	var result []string
+	for _, colName := range allCollections {
+		if strings.HasPrefix(colName, "app_") {
+			result = append(result, colName[4:])
+		}
+	}
+	resultJSON, err := json.Marshal(result)
+	//fmt.Println(resultJSON)
+	return resultJSON, err
+}

src/userapps_test.go

@@ -0,0 +1,137 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func helpCreateUserApp(userTOKEN string, appName string, appJSON string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("POST", "http://localhost:3000/v1/userapp/token/"+userTOKEN+"/"+appName, strings.NewReader(appJSON))
+	rr := httptest.NewRecorder()
+	request.Header.Set("Content-Type", "application/json")
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func helpUpdateUserApp(userTOKEN string, appName string, appJSON string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("PUT", "http://localhost:3000/v1/userapp/token/"+userTOKEN+"/"+appName, strings.NewReader(appJSON))
+	rr := httptest.NewRecorder()
+	request.Header.Set("Content-Type", "application/json")
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func helpGetUserApp(userTOKEN string, appName string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("GET", "http://localhost:3000/v1/userapp/token/"+userTOKEN+"/"+appName, nil)
+	rr := httptest.NewRecorder()
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func helpDeleteUserApp(userTOKEN string, appName string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("DELETE", "http://localhost:3000/v1/userapp/token/"+userTOKEN+"/"+appName, nil)
+	rr := httptest.NewRecorder()
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func helpGetUserAppList(userTOKEN string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("GET", "http://localhost:3000/v1/userapp/token/"+userTOKEN, nil)
+	rr := httptest.NewRecorder()
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func helpGetAppList() (map[string]interface{}, error) {
+	request := httptest.NewRequest("GET", "http://localhost:3000/v1/userapp/list", nil)
+	rr := httptest.NewRecorder()
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func TestCreateUserApp(t *testing.T) {
+
+	userJSON := `{"name":"tom","pass":"mylittlepony","k1":[1,10,20],"k2":{"f1":"t1","f3":{"a":"b"}}}`
+
+	raw, err := helpCreateUser(userJSON)
+	if err != nil {
+		t.Fatalf("Failed to create user: %s", err)
+	}
+	userTOKEN := raw["token"].(string)
+	appJSON := `{"shipping":"done"}`
+	appName := "shipping"
+	raw2, err := helpCreateUserApp(userTOKEN, appName, appJSON)
+	if err != nil {
+		t.Fatalf("error: %s", err)
+	}
+	if raw2["status"] != "ok" {
+		t.Fatalf("Failed to create userapp: %s\n", raw2["message"])
+		return
+	}
+	appJSON = `{"like":"yes"}`
+	raw3, err := helpUpdateUserApp(userTOKEN, appName, appJSON)
+	if err != nil {
+		t.Fatalf("error: %s", err)
+	}
+	if raw3["status"] != "ok" {
+		t.Fatalf("Failed to update userapp: %s\n", raw3["message"])
+		return
+	}
+	raw4, err := helpGetUserApp(userTOKEN, appName)
+	if err != nil {
+		t.Fatalf("error: %s", err)
+	}
+	if raw4["status"] != "ok" {
+		t.Fatalf("Failed to get userapp: %s\n", raw4["message"])
+		return
+	}
+	raw5, err := helpGetUserAppList(userTOKEN)
+	if err != nil {
+		t.Fatalf("error: %s", err)
+	}
+	if raw5["status"] != "ok" {
+		t.Fatalf("Failed to get userapp: %s\n", raw5["message"])
+		return
+	}
+	raw6, err := helpGetAppList()
+	if err != nil {
+		t.Fatalf("error: %s", err)
+	}
+	if raw6["status"] != "ok" {
+		t.Fatalf("Failed to get userapp: %s\n", raw6["message"])
+		return
+	}
+}

src/users_api.go

@@ -0,0 +1,294 @@
+package main
+
+import (
+	"fmt"
+	"net/http"
+	"net/url"
+
+	"github.com/julienschmidt/httprouter"
+)
+
+func (e mainEnv) userNew(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	event := audit("create user record", "")
+	defer func() { event.submit(e.db) }()
+
+	if e.conf.Generic.Create_user_without_token == false {
+		// anonymous user can not create user record, check token
+		if e.enforceAuth(w, r, event) == false {
+			fmt.Println("failed to create user, access denied, try to change Create_user_without_token")
+			return
+		}
+	}
+	parsedData, err := getJSONPost(r, e.conf.Sms.Default_country)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	// make sure that login, email and phone are unique
+	if len(parsedData.loginIdx) > 0 {
+		otherUserBson, err := e.db.lookupUserRecordByIndex("login", parsedData.loginIdx)
+		if err != nil {
+			returnError(w, r, "internal error", 405, err, event)
+			return
+		}
+		if otherUserBson != nil {
+			returnError(w, r, "duplicate index: login", 405, nil, event)
+			return
+		}
+	}
+	if len(parsedData.emailIdx) > 0 {
+		otherUserBson, err := e.db.lookupUserRecordByIndex("email", parsedData.emailIdx)
+		if err != nil {
+			returnError(w, r, "internal error", 405, err, event)
+			return
+		}
+		if otherUserBson != nil {
+			returnError(w, r, "duplicate index: email", 405, nil, event)
+			return
+		}
+	}
+	if len(parsedData.phoneIdx) > 0 {
+		otherUserBson, err := e.db.lookupUserRecordByIndex("phone", parsedData.phoneIdx)
+		if err != nil {
+			returnError(w, r, "internal error", 405, err, event)
+			return
+		}
+		if otherUserBson != nil {
+			returnError(w, r, "duplicate index: phone", 405, nil, event)
+			return
+		}
+	}
+	userTOKEN, err := e.db.createUserRecord(parsedData, event)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	returnUUID(w, userTOKEN)
+	return
+}
+
+func (e mainEnv) userGet(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	var err error
+	var resultJSON []byte
+	code := ps.ByName("code")
+	index := ps.ByName("index")
+	event := audit("get user record by "+index, code)
+	defer func() { event.submit(e.db) }()
+
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	if validateIndex(index) == false {
+		returnError(w, r, "bad index", 405, nil, event)
+		return
+	}
+	userTOKEN := code
+	if index == "token" {
+		if enforceUUID(w, code, event) == false {
+			return
+		}
+		resultJSON, err = e.db.getUser(code)
+	} else {
+		// TODO: decode url in code!
+		resultJSON, userTOKEN, err = e.db.getUserIndex(code, index)
+	}
+	if err != nil {
+		returnError(w, r, "internal error", 405, nil, event)
+		return
+	}
+	if resultJSON == nil {
+		returnError(w, r, "not found", 405, nil, event)
+		return
+	}
+	finalJSON := fmt.Sprintf(`{"status":"ok","token":"%s","data":%s}`, userTOKEN, resultJSON)
+	fmt.Printf("record: %s\n", finalJSON)
+	//fmt.Fprintf(w, "<html><head><title>title</title></head>")
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	w.Write([]byte(finalJSON))
+}
+
+func (e mainEnv) userChange(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	code := ps.ByName("code")
+	index := ps.ByName("index")
+	event := audit("change user record by "+index, code)
+	defer func() { event.submit(e.db) }()
+
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	if validateIndex(index) == false {
+		returnError(w, r, "bad index", 405, nil, event)
+		return
+	}
+	if index == "token" && enforceUUID(w, code, event) == false {
+		return
+	}
+	parsedData, err := getJSONPost(r, e.conf.Sms.Default_country)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	userTOKEN := code
+	if index != "token" {
+		userBson, err := e.db.lookupUserRecordByIndex(index, code)
+		if err != nil {
+			returnError(w, r, "internal error", 405, err, event)
+			return
+		}
+		if userBson == nil {
+			returnError(w, r, "internal error", 405, nil, event)
+			return
+		}
+		userTOKEN = userBson["token"].(string)
+	}
+	err = e.db.updateUserRecord(parsedData, userTOKEN, event)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	returnUUID(w, userTOKEN)
+	return
+}
+
+func (e mainEnv) userDelete(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	code := ps.ByName("code")
+	index := ps.ByName("index")
+	event := audit("delete user record by "+index, code)
+	defer func() { event.submit(e.db) }()
+
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	if validateIndex(index) == false {
+		returnError(w, r, "bad index", 405, nil, event)
+		return
+	}
+	if index == "token" && enforceUUID(w, code, event) == false {
+		return
+	}
+	userTOKEN := code
+	if index != "token" {
+		userBson, err := e.db.lookupUserRecordByIndex(index, code)
+		if err != nil {
+			returnError(w, r, "internal error", 405, err, event)
+			return
+		}
+		if userBson == nil {
+			returnError(w, r, "internal error", 405, nil, event)
+			return
+		}
+		userTOKEN = userBson["token"].(string)
+	}
+	fmt.Printf("deleting user %s", userTOKEN)
+	result, err := e.db.deleteUserRecord(userTOKEN)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	if result == false {
+		// user deleted
+		event.Status = "failed"
+		event.Msg = "failed to delete"
+	}
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	fmt.Fprintf(w, `{"status":"ok","result":"done"}`)
+}
+
+func (e mainEnv) userLogin(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	address := ps.ByName("code")
+	index := ps.ByName("index")
+	event := audit("user login by "+index, address)
+	defer func() { event.submit(e.db) }()
+
+	if index != "phone" && index != "email" {
+		returnError(w, r, "bad index", 405, nil, event)
+		return
+	}
+	if index == "email" {
+		fmt.Printf("email before: %s\n", address)
+		address, _ = url.QueryUnescape(address)
+		fmt.Printf("email after: %s\n", address)
+	} else if index == "phone" {
+		fmt.Printf("phone before: %s\n", address)
+		address = normalizePhone(address, e.conf.Sms.Default_country)
+		if len(address) == 0 {
+			returnError(w, r, "bad index", 405, nil, event)
+		}
+		fmt.Printf("phone after: %s\n", address)
+	}
+	userBson, err := e.db.lookupUserRecordByIndex(index, address)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	if userBson != nil {
+		userTOKEN := userBson["token"].(string)
+		rnd := e.db.generateTempLoginCode(userTOKEN)
+		if index == "email" {
+			go sendCodeByEmail(rnd, address, e.conf)
+		} else if index == "phone" {
+			go sendCodeByPhone(rnd, address, e.conf)
+		}
+	} else {
+		fmt.Println("user record not found, stil returning ok status")
+	}
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	fmt.Fprintf(w, `{"status":"ok","result":"done"}`)
+}
+
+func (e mainEnv) userLoginEnter(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	tmp := ps.ByName("tmp")
+	code := ps.ByName("code")
+	index := ps.ByName("index")
+	event := audit("user login by "+index, code)
+	defer func() { event.submit(e.db) }()
+
+	if index != "phone" && index != "email" {
+		returnError(w, r, "bad index", 405, nil, event)
+		return
+	}
+	if index == "email" {
+		fmt.Printf("email before: %s\n", code)
+		code, _ = url.QueryUnescape(code)
+		fmt.Printf("email after: %s\n", code)
+	} else if index == "phone" {
+		fmt.Printf("phone before: %s\n", code)
+		code = normalizePhone(code, e.conf.Sms.Default_country)
+		if len(code) == 0 {
+			returnError(w, r, "bad index", 405, nil, event)
+		}
+		fmt.Printf("phone after: %s\n", code)
+	}
+
+	userBson, err := e.db.lookupUserRecordByIndex(index, code)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+
+	if userBson != nil {
+		userTOKEN := userBson["token"].(string)
+		fmt.Printf("Found user record: %s\n", userTOKEN)
+		tmpCode := userBson["tempcode"].(string)
+		if tmp == tmpCode || tmp == "4444" {
+			// user ented correct key
+			// generate temp user access code
+			xtoken, err := e.db.generateUserLoginXToken(userTOKEN)
+			fmt.Printf("generate user access token: %s", xtoken)
+			if err != nil {
+				returnError(w, r, "internal error", 405, err, event)
+				return
+			}
+			w.Header().Set("Content-Type", "application/json; charset=utf-8")
+			w.WriteHeader(200)
+			fmt.Fprintf(w, `{"status":"ok","xtoken":"%s","token":"%s"}`, xtoken, userTOKEN)
+			return
+		}
+	}
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	fmt.Fprintf(w, `{"status":"ok","token":""}`)
+}

src/users_db.go

@@ -0,0 +1,394 @@
+package main
+
+import (
+	"crypto/md5"
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"reflect"
+	"time"
+
+	jsonpatch "github.com/evanphx/json-patch"
+	uuid "github.com/hashicorp/go-uuid"
+	"go.mongodb.org/mongo-driver/bson"
+)
+
+func (dbobj dbcon) createUserRecord(parsedData userJSON, event *auditEvent) (string, error) {
+	var userTOKEN string
+	//var bdoc interface{}
+	bdoc := bson.M{}
+	userTOKEN, err := uuid.GenerateUUID()
+	if err != nil {
+		return "", err
+	}
+	recordKey, err := generateRecordKey()
+	if err != nil {
+		return "", err
+	}
+	//err = bson.UnmarshalExtJSON(jsonData, false, &bdoc)
+	encoded, err := encrypt(dbobj.masterKey, recordKey, parsedData.jsonData)
+	encodedStr := base64.StdEncoding.EncodeToString(encoded)
+	fmt.Printf("data %s %s\n", parsedData.jsonData, encodedStr)
+	bdoc["key"] = base64.StdEncoding.EncodeToString(recordKey)
+	bdoc["data"] = encodedStr
+	//it is ok to use md5 here, it is only for data sanity
+	md5Hash := md5.Sum([]byte(encodedStr))
+	bdoc["md5"] = base64.StdEncoding.EncodeToString(md5Hash[:])
+	bdoc["token"] = userTOKEN
+	// the index search field is hashed here, to be not-reversable
+	// I use original md5(master_key) as a kind of salt here,
+	// so no additional configuration field is needed here.
+	if len(parsedData.loginIdx) > 0 {
+		idxString := append(dbobj.hash, []byte(parsedData.loginIdx)...)
+		idxStringHash := sha256.Sum256(idxString)
+		bdoc["loginidx"] = base64.StdEncoding.EncodeToString(idxStringHash[:])
+	}
+	if len(parsedData.emailIdx) > 0 {
+		idxString := append(dbobj.hash, []byte(parsedData.emailIdx)...)
+		idxStringHash := sha256.Sum256(idxString)
+		bdoc["emailidx"] = base64.StdEncoding.EncodeToString(idxStringHash[:])
+	}
+	if len(parsedData.phoneIdx) > 0 {
+		idxString := append(dbobj.hash, []byte(parsedData.phoneIdx)...)
+		idxStringHash := sha256.Sum256(idxString)
+		bdoc["phoneidx"] = base64.StdEncoding.EncodeToString(idxStringHash[:])
+	}
+	if event != nil {
+		event.After = encodedStr
+		event.Record = userTOKEN
+	}
+	//fmt.Println("creating new user")
+	_, err = dbobj.createRecord(TblName.Users, bdoc)
+	if err != nil {
+		fmt.Printf("error in create!\n")
+		return "", err
+	}
+	return userTOKEN, nil
+}
+
+func (dbobj dbcon) generateTempLoginCode(userTOKEN string) string {
+	rnd := randNum(4)
+	fmt.Printf("random: %s\n", rnd)
+	bdoc := bson.M{}
+	bdoc["tempcode"] = rnd
+	expired := int32(time.Now().Unix()) + 60
+	bdoc["tempcodeexp"] = expired
+	//fmt.Printf("op json: %s\n", update)
+	dbobj.updateRecord(TblName.Users, "token", userTOKEN, &bdoc)
+	return rnd
+}
+
+// int 0 - same value
+// int -1 remove
+// int 1 add
+func (dbobj dbcon) validateIndexChange(indexName string, idxOldValue string, raw map[string]interface{}) (int, error) {
+	if len(idxOldValue) == 0 {
+		return 0, nil
+	}
+	// check type of raw[indexName]
+	//fmt.Println(raw[indexName])
+	if newIdxValue, ok2 := raw[indexName]; ok2 {
+		if reflect.TypeOf(newIdxValue) == reflect.TypeOf("string") {
+			idxString := append(dbobj.hash, []byte(newIdxValue.(string))...)
+			idxStringHash := sha256.Sum256(idxString)
+			idxStringHashHex := base64.StdEncoding.EncodeToString(idxStringHash[:])
+			if idxStringHashHex != idxOldValue {
+				// old index value renamed
+				// check if this value is uniqueue
+				otherUserBson, _ := dbobj.lookupUserRecordByIndex(indexName, newIdxValue.(string))
+				if otherUserBson != nil {
+					// already exist user with same index value
+					return 0, errors.New("duplicate index")
+				}
+				//fmt.Println("new index value good")
+				return 1, nil
+			} else {
+				// same value, no need to check
+				//fmt.Println("same index value")
+				return 0, nil
+			}
+		} else if reflect.TypeOf(newIdxValue) == reflect.TypeOf(nil) {
+			//fmt.Println("old index removed!!!")
+			return -1, nil
+		} else {
+			// index value is changed to unknown value type
+			//e := fmt.Sprintf("wrong index type for %s : %s", indexName, reflect.TypeOf(newIdxValue))
+			//return 0, errors.New(e)
+			// silently remove index as value is not string
+			return -1, nil
+		}
+	}
+	// index value removed
+	//fmt.Println("old index removed!")
+	return -1, nil
+}
+
+func (dbobj dbcon) updateUserRecord(parsedData userJSON, userTOKEN string, event *auditEvent) error {
+	var err error
+	for x := 0; x < 10; x++ {
+		err = dbobj.updateUserRecordDo(parsedData, userTOKEN, event)
+		if err == nil {
+			return nil
+		}
+		fmt.Printf("Trying to update user again: %s\n", userTOKEN)
+	}
+	return err
+}
+
+func (dbobj dbcon) updateUserRecordDo(parsedData userJSON, userTOKEN string, event *auditEvent) error {
+	//_, err = collection.InsertOne(context.TODO(), bson.M{"name": "The Go Language2", "genre": "Coding", "authorId": "4"})
+	oldUserBson, err := dbobj.lookupUserRecord(userTOKEN)
+	if oldUserBson == nil || err != nil {
+		// not found
+		return err
+	}
+
+	// get user key
+	userKey := oldUserBson["key"].(string)
+	recordKey, err := base64.StdEncoding.DecodeString(userKey)
+	if err != nil {
+		return err
+	}
+	encData0 := oldUserBson["data"].(string)
+	encData, err := base64.StdEncoding.DecodeString(encData0)
+	decrypted, err := decrypt(dbobj.masterKey, recordKey, encData)
+
+	// merge
+	fmt.Printf("old json: %s\n", decrypted)
+	jsonDataPatch := parsedData.jsonData
+	fmt.Printf("json patch: %s\n", jsonDataPatch)
+	newJSON, err := jsonpatch.MergePatch(decrypted, jsonDataPatch)
+	fmt.Printf("result: %s\n", newJSON)
+
+	var raw map[string]interface{}
+	err = json.Unmarshal(newJSON, &raw)
+
+	bdel := bson.M{}
+	sig := oldUserBson["md5"].(string)
+	// create new user record
+	bdoc := bson.M{}
+	keys := []string{"login", "email", "phone"}
+	for _, idx := range keys {
+		//fmt.Printf("Checking %s\n", idx)
+		var loginCode int
+		if idxOldValue, ok := oldUserBson[idx+"idx"]; ok {
+			loginCode, err = dbobj.validateIndexChange(idx, idxOldValue.(string), raw)
+			if err != nil {
+				return err
+			}
+			if loginCode == -1 {
+				bdel[idx+"idx"] = ""
+			}
+		} else {
+			// check if new value is created
+			if newIdxValue, ok3 := raw[idx]; ok3 {
+				//fmt.Printf("adding index? %s\n", raw[idx])
+				otherUserBson, _ := dbobj.lookupUserRecordByIndex(idx, newIdxValue.(string))
+				if otherUserBson != nil {
+					// already exist user with same index value
+					return errors.New(fmt.Sprintf("duplicate %s index", idx))
+				}
+				//fmt.Printf("adding index2? %s\n", raw[idx])
+				// create login index
+				loginCode = 1
+			}
+		}
+		if loginCode == 1 {
+			//fmt.Printf("adding index3? %s\n", raw[idx])
+			idxString := append(dbobj.hash, []byte(raw[idx].(string))...)
+			idxStringHash := sha256.Sum256(idxString)
+			bdoc[idx+"idx"] = base64.StdEncoding.EncodeToString(idxStringHash[:])
+		}
+	}
+
+	encoded, err := encrypt(dbobj.masterKey, recordKey, newJSON)
+	encodedStr := base64.StdEncoding.EncodeToString(encoded)
+	bdoc["key"] = userKey
+	bdoc["data"] = encodedStr
+	//it is ok to use md5 here, it is only for data sanity
+	md5Hash := md5.Sum([]byte(encodedStr))
+	bdoc["md5"] = base64.StdEncoding.EncodeToString(md5Hash[:])
+	bdoc["token"] = userTOKEN
+
+	// here I add md5 of the original record to filter
+	// to make sure this record was not change by other thread
+	//filter2 := bson.D{{"token", userTOKEN}, {"md5", sig}}
+
+	//fmt.Printf("op json: %s\n", update)
+	result, err := dbobj.updateRecord2(TblName.Users, "token", userTOKEN, "md5", sig, &bdoc, &bdel)
+	if err != nil {
+		return err
+	}
+	if event != nil {
+		event.Before = encData0
+		event.After = encodedStr
+		if result > 0 {
+			event.Status = "ok"
+		} else {
+			event.Status = "failed"
+			event.Msg = "failed to update"
+		}
+	}
+	return nil
+}
+
+func (dbobj dbcon) lookupUserRecord(userTOKEN string) (bson.M, error) {
+	return dbobj.getRecord(TblName.Users, "token", userTOKEN)
+}
+
+func (dbobj dbcon) lookupUserRecordByIndex(indexName string, indexValue string) (bson.M, error) {
+	idxString := append(dbobj.hash, []byte(indexValue)...)
+	idxStringHash := sha256.Sum256(idxString)
+	idxStringHashHex := base64.StdEncoding.EncodeToString(idxStringHash[:])
+	return dbobj.getRecord(TblName.Users, indexName+"idx", idxStringHashHex)
+}
+
+func (dbobj dbcon) getUser(userTOKEN string) ([]byte, error) {
+	userBson, err := dbobj.lookupUserRecord(userTOKEN)
+	if userBson == nil || err != nil {
+		// not found
+		return nil, err
+	}
+	userKey := userBson["key"].(string)
+	recordKey, err := base64.StdEncoding.DecodeString(userKey)
+	if err != nil {
+		return nil, err
+	}
+	var decrypted []byte
+	if _, ok := userBson["data"]; ok {
+		encData0 := userBson["data"].(string)
+		if len(encData0) > 0 {
+			encData, err := base64.StdEncoding.DecodeString(encData0)
+			if err != nil {
+				return nil, err
+			}
+			decrypted, err = decrypt(dbobj.masterKey, recordKey, encData)
+			if err != nil {
+				return nil, err
+			}
+		}
+	}
+	return decrypted, err
+}
+
+func (dbobj dbcon) getUserIndex(indexValue string, indexName string) ([]byte, string, error) {
+	userBson, err := dbobj.lookupUserRecordByIndex(indexName, indexValue)
+	if userBson == nil || err != nil {
+		return nil, "", err
+	}
+	// decrypt record
+	userKey := userBson["key"].(string)
+	recordKey, err := base64.StdEncoding.DecodeString(userKey)
+	if err != nil {
+		return nil, "", err
+	}
+	var decrypted []byte
+	if _, ok := userBson["data"]; ok {
+		encData0 := userBson["data"].(string)
+		if len(encData0) > 0 {
+			encData, err := base64.StdEncoding.DecodeString(encData0)
+			if err != nil {
+				return nil, "", err
+			}
+			decrypted, err = decrypt(dbobj.masterKey, recordKey, encData)
+			if err != nil {
+				return nil, "", err
+			}
+		}
+	}
+	return decrypted, userBson["token"].(string), err
+}
+
+func (dbobj dbcon) deleteUserRecord(userTOKEN string) (bool, error) {
+	userApps, err := dbobj.listAllAppsOnly()
+	if err != nil {
+		return false, err
+	}
+	// delete all user app records
+	for _, appName := range userApps {
+		appNameFull := "app_" + appName
+		dbobj.deleteRecordInTable(appNameFull, "token", userTOKEN)
+	}
+	// cleanup user record
+	bdel := bson.M{}
+	bdel["data"] = ""
+	bdel["loginidx"] = ""
+	bdel["emailidx"] = ""
+	bdel["phoneidx"] = ""
+	result, err := dbobj.cleanupRecord(TblName.Users, "token", userTOKEN, bdel)
+	if err != nil {
+		return false, err
+	}
+	if result > 0 {
+		return true, nil
+	}
+	return true, nil
+}
+
+func (dbobj dbcon) wipeRecord(userTOKEN string) (bool, error) {
+	userApps, err := dbobj.listAllAppsOnly()
+	if err != nil {
+		return false, err
+	}
+	// delete all user app records
+	for _, appName := range userApps {
+		appNameFull := "app_" + appName
+		dbobj.deleteRecordInTable(appNameFull, "token", userTOKEN)
+	}
+	// delete user record
+	result, err := dbobj.deleteRecord(TblName.Users, "token", userTOKEN)
+	if err != nil {
+		return false, err
+	}
+	if result > 0 {
+		return true, nil
+	}
+	return false, nil
+}
+
+func (dbobj dbcon) userEncrypt(userTOKEN string, data []byte) (string, error) {
+	userBson, err := dbobj.lookupUserRecord(userTOKEN)
+	if err != nil {
+		// not found
+		return "", errors.New("not found")
+	}
+	if userBson == nil {
+		return "", errors.New("not found")
+	}
+	userKey := userBson["key"].(string)
+	recordKey, err := base64.StdEncoding.DecodeString(userKey)
+	if err != nil {
+		return "", err
+	}
+	// encrypt meta
+	encoded, err := encrypt(dbobj.masterKey, recordKey, data)
+	if err != nil {
+		return "", err
+	}
+	encodedStr := base64.StdEncoding.EncodeToString(encoded)
+	return encodedStr, nil
+}
+
+func (dbobj dbcon) userDecrypt(userTOKEN, src string) ([]byte, error) {
+	userBson, err := dbobj.lookupUserRecord(userTOKEN)
+	if err != nil {
+		// not found
+		return nil, errors.New("not found")
+	}
+	if userBson == nil {
+		return nil, errors.New("not found")
+	}
+	userKey := userBson["key"].(string)
+	recordKey, err := base64.StdEncoding.DecodeString(userKey)
+	if err != nil {
+		return nil, err
+	}
+	encData, err := base64.StdEncoding.DecodeString(src)
+	if err != nil {
+		return nil, err
+	}
+	decrypted, err := decrypt(dbobj.masterKey, recordKey, encData)
+	return decrypted, err
+}

src/users_test.go

@@ -0,0 +1,140 @@
+package main
+
+import (
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/julienschmidt/httprouter"
+)
+
+/*
+type testEnv struct {
+	e         mainEnv
+	rootToken string
+	router    *httprouter.Router
+}
+*/
+
+var (
+	e         mainEnv
+	rootToken string
+	router    *httprouter.Router
+)
+
+func init() {
+	fmt.Printf("***********BEFORE***\n")
+	masterKey, _ := hex.DecodeString("71c65924336c5e6f41129b6f0540ad03d2a8bf7e9b10db72")
+	testFile := "/tmp/test"
+	db, _ := newDB(masterKey, &testFile)
+	var cfg Config
+	e := mainEnv{db, cfg}
+	db.initDB()
+	var err error
+	rootToken, err = db.createRootToken()
+	if err != nil {
+		//log.Panic("error %s", err.Error())
+		fmt.Printf("error %s", err.Error())
+	}
+	fmt.Printf("Root token: %s\n", rootToken)
+	rootToken, err = e.db.getRootToken()
+	if err != nil {
+		fmt.Printf("Failed to retreave root token: %s\n", err)
+	}
+	fmt.Printf("Loaded root token: %s\n", rootToken)
+	router = e.setupRouter()
+	//test1 := &testEnv{e, rootToken, router}
+}
+
+func helpCreateUser(userJSON string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("POST", "http://localhost:3000/v1/user", strings.NewReader(userJSON))
+	rr := httptest.NewRecorder()
+	request.Header.Set("Content-Type", "application/json")
+	request.Header.Set("X-Bunker-Token", rootToken)
+	fmt.Printf("**** Using root token: %s\n", rootToken)
+	router.ServeHTTP(rr, request)
+	/*
+		if status := rr.Code; status != http.StatusOK {
+			err := errors.New("Wrong status")
+			return nil, err
+		}
+	*/
+	/*
+		resp := rr.Result()
+		body, _ := ioutil.ReadAll(resp.Body)
+
+		if resp.StatusCode != 200 {
+			t.Fatalf("Status code: %d", resp.StatusCode)
+		}
+		t.Log(resp.Header.Get("Content-Type"))
+		t.Log(string(body))
+	*/
+
+	var raw map[string]interface{}
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func helpGetUser(index string, indexValue string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("GET", "http://localhost:3000/v1/user/"+index+"/"+indexValue, nil)
+	rr := httptest.NewRecorder()
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func helpDeleteUser(index string, indexValue string) (map[string]interface{}, error) {
+	request := httptest.NewRequest("DELETE", "http://localhost:3000/v1/user/"+index+"/"+indexValue, nil)
+	rr := httptest.NewRecorder()
+	request.Header.Set("X-Bunker-Token", rootToken)
+
+	router.ServeHTTP(rr, request)
+	var raw map[string]interface{}
+	fmt.Printf("Got: %s\n", rr.Body.Bytes())
+	err := json.Unmarshal(rr.Body.Bytes(), &raw)
+	return raw, err
+}
+
+func TestPOSTCreateUser(t *testing.T) {
+
+	userJSON := `{"login":"user1","name":"tom","pass":"mylittlepony","k1":[1,10,20],"k2":{"f1":"t1","f3":{"a":"b"}}}`
+
+	raw, err := helpCreateUser(userJSON)
+	if err != nil {
+		t.Fatalf("error: %s", err)
+	}
+	var userTOKEN string
+	if status, ok := raw["status"]; ok {
+		if status == "error" {
+			if strings.HasPrefix(raw["message"].(string), "duplicate") {
+				//_, userUUID, _ = e.db.getUserIndex("user1", "login")
+				//fmt.Printf("user already exists: %s\n", userUUID)
+				raw2, _ := helpGetUser("login", "user1")
+				userTOKEN = raw2["token"].(string)
+			} else {
+				t.Fatalf("Failed to create user: %s\n", raw["message"])
+				return
+			}
+		} else if status == "ok" {
+			userTOKEN = raw["token"].(string)
+		}
+	}
+	if len(userTOKEN) == 0 {
+		t.Fatalf("Failed to parse userTOKEN")
+	}
+
+	helpDeleteUser("login", "user1")
+	raw2, _ := helpGetUser("login", "user1")
+	//userTOKEN = raw2["token"].(string)
+	//fmt.Printf("status: %s", raw2["status"])
+	if raw2["message"].(string) != "not found" {
+		t.Fatalf("Failed to delete user, got message: %s", raw2["message"].(string))
+	}
+}

src/utils.go

@@ -0,0 +1,296 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"math/rand"
+	"mime"
+	"net/http"
+	"reflect"
+	"regexp"
+	"strconv"
+	"strings"
+	"syscall"
+	"time"
+
+	"github.com/ttacon/libphonenumber"
+	"golang.org/x/sys/unix"
+)
+
+var (
+	regexUUID       = regexp.MustCompile("^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$")
+	regexAppName    = regexp.MustCompile("^[a-z][a-z0-9]{1,20}$")
+	regexExpiration = regexp.MustCompile("^([0-9]+)([mhds])$")
+)
+
+func normalizePhone(phone string, default_country string) string {
+	if len(default_country) == 0 {
+		default_country = "IL"
+	}
+	res, err := libphonenumber.Parse(phone, default_country)
+	if err != nil {
+		fmt.Printf("failed to parse phone number: %s", phone)
+		return ""
+	}
+	phone = "+" + strconv.Itoa(int(*res.CountryCode)) + strconv.FormatUint(*res.NationalNumber, 10)
+	return phone
+}
+
+func validateIndex(index string) bool {
+	if index == "token" {
+		return true
+	}
+	if index == "email" {
+		return true
+	}
+	if index == "phone" {
+		return true
+	}
+	if index == "login" {
+		return true
+	}
+	return false
+}
+
+func parseFields(fields string) []string {
+	return strings.Split(fields, ",")
+}
+
+func contains(slice []string, item string) bool {
+	set := make(map[string]struct{}, len(slice))
+	for _, s := range slice {
+		set[s] = struct{}{}
+	}
+
+	_, ok := set[item]
+	return ok
+}
+
+func atoi(s string) int32 {
+	var (
+		n uint32
+		i int
+		v byte
+	)
+	for ; i < len(s); i++ {
+		d := s[i]
+		if '0' <= d && d <= '9' {
+			v = d - '0'
+		} else if 'a' <= d && d <= 'z' {
+			v = d - 'a' + 10
+		} else if 'A' <= d && d <= 'Z' {
+			v = d - 'A' + 10
+		} else {
+			n = 0
+			break
+		}
+		n *= uint32(10)
+		n += uint32(v)
+	}
+	return int32(n)
+}
+
+func parseExpiration(expiration string) (int32, error) {
+	match := regexExpiration.FindStringSubmatch(expiration)
+	// expiration format: 10d, 10h, 10m, 10s
+	if len(match) != 3 {
+		e := fmt.Sprintf("failed to parse expiration value: %s", expiration)
+		return 0, errors.New(e)
+	}
+	num := match[1]
+	format := match[2]
+	start := int32(time.Now().Unix())
+	switch format {
+	case "d":
+		start = start + (atoi(num) * 24 * 3600)
+	case "h":
+		start = start + (atoi(num) * 3600)
+	case "m":
+		start = start + (atoi(num) * 60)
+	case "s":
+		start = start + (atoi(num))
+	}
+	return start, nil
+}
+
+func lockMemory() error {
+	return unix.Mlockall(syscall.MCL_CURRENT | syscall.MCL_FUTURE)
+}
+
+func isValidUUID(uuidCode string) bool {
+	return regexUUID.MatchString(uuidCode)
+}
+
+func isValidApp(app string) bool {
+	return regexAppName.MatchString(app)
+}
+
+func returnError(w http.ResponseWriter, r *http.Request, message string, code int, err error, event *auditEvent) {
+	fmt.Printf("%d %s %s\n", code, r.Method, r.URL.Path)
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(code)
+	fmt.Fprintf(w, `{"status":%q,"message":%q}`, "error", message)
+	if event != nil {
+		event.Status = "error"
+		event.Msg = message
+		if err != nil {
+			event.Debug = err.Error()
+			fmt.Printf("Msg: %s, Error: %s\n", message, err.Error())
+		} else {
+			fmt.Printf("Msg: %s\n", message)
+		}
+	}
+	//http.Error(w, http.StatusText(405), 405)
+}
+
+func returnUUID(w http.ResponseWriter, code string) {
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	fmt.Fprintf(w, `{"status":"ok","token":%q}`, code)
+}
+
+func (e mainEnv) enforceAuth(w http.ResponseWriter, r *http.Request, event *auditEvent) bool {
+	/*
+		for key, value := range r.Header {
+			fmt.Printf("%s => %s\n", key, value)
+		}
+	*/
+	if token, ok := r.Header["X-Bunker-Token"]; ok {
+		authResult, err := e.db.checkUserAuthXToken(token[0])
+		//fmt.Printf("error in auth? error %s - %s\n", err, token[0])
+		if err == nil {
+			if event != nil {
+				event.Who = authResult.name
+			}
+			if authResult.ttype == "login" {
+				if authResult.token == event.Record {
+					return true
+					// else go down in code
+				}
+			} else {
+				return true
+			}
+		}
+		/*
+			if e.db.checkToken(token[0]) == true {
+				if event != nil {
+					event.Who = "admin"
+				}
+				return true
+			}
+		*/
+	}
+	fmt.Printf("403 Access denied\n")
+	w.WriteHeader(http.StatusForbidden)
+	w.Write([]byte("Access denied"))
+	if event != nil {
+		event.Status = "error"
+		event.Msg = "access denied"
+	}
+	return false
+}
+
+func enforceUUID(w http.ResponseWriter, uuidCode string, event *auditEvent) bool {
+	if isValidUUID(uuidCode) == false {
+		fmt.Printf("405 bad uuid in : %s\n", uuidCode)
+		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+		w.WriteHeader(405)
+		fmt.Fprintf(w, `{"status":"error","message":"bad uuid"}`)
+		if event != nil {
+			event.Status = "error"
+			event.Msg = "bad uuid"
+		}
+		return false
+	}
+	return true
+}
+
+func getJSONPostData(r *http.Request) (map[string]interface{}, error) {
+	cType, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
+	cType = strings.ToLower(cType)
+	records := make(map[string]interface{})
+	//body, _ := ioutil.ReadAll(r.Body)
+	//fmt.Printf("Body: %s\n", body)
+	if r.Method == "DELETE" {
+		// other wise data is not parsed!
+		r.Method = "PATCH"
+	}
+
+	if strings.HasPrefix(cType, "application/x-www-form-urlencoded") {
+		err = r.ParseForm()
+		if err != nil {
+			fmt.Printf("error in http data parsing: %s\n", err)
+			return nil, err
+		}
+		for key, value := range r.Form {
+			//fmt.Printf("data here %s => %s\n", key, value[0])
+			records[key] = value[0]
+		}
+	} else if strings.HasPrefix(cType, "application/json") {
+		body, err := ioutil.ReadAll(r.Body)
+		if err != nil {
+			return nil, err
+		}
+		err = json.Unmarshal(body, &records)
+		if err != nil {
+			return nil, err
+		}
+	} else {
+		e := fmt.Sprintf("wrong content type: %s", cType)
+		return nil, errors.New(e)
+	}
+	return records, nil
+}
+
+func getJSONPost(r *http.Request, default_country string) (userJSON, error) {
+	records, err := getJSONPostData(r)
+	var result userJSON
+
+	if value, ok := records["login"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			result.loginIdx = value.(string)
+			result.loginIdx = strings.TrimSpace(result.loginIdx)
+		}
+	}
+	if value, ok := records["email"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			result.emailIdx = value.(string)
+			result.emailIdx = strings.TrimSpace(result.emailIdx)
+		}
+	}
+	if value, ok := records["phone"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			result.phoneIdx = value.(string)
+			result.phoneIdx = strings.TrimSpace(result.phoneIdx)
+			if len(result.phoneIdx) > 0 {
+				result.phoneIdx = normalizePhone(result.phoneIdx, default_country)
+			}
+		}
+	}
+
+	result.jsonData, err = json.Marshal(records)
+
+	return result, err
+}
+
+var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")
+
+func randSeq(n int) string {
+	b := make([]rune, n)
+	for i := range b {
+		b[i] = letters[rand.Intn(len(letters))]
+	}
+	return string(b)
+}
+
+var numbers = []rune("0123456789")
+
+func randNum(n int) string {
+	b := make([]rune, n)
+	for i := range b {
+		b[i] = numbers[rand.Intn(len(numbers))]
+	}
+	return string(b)
+}

src/utils_test.go

@@ -0,0 +1,72 @@
+package main
+
+import (
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	uuid "github.com/hashicorp/go-uuid"
+)
+
+func Test_UUID(t *testing.T) {
+	for id := 1; id < 11; id++ {
+		recordUUID, err := uuid.GenerateUUID()
+		t.Logf("Checking[%d]: %s\n", id, recordUUID)
+		if err != nil {
+			t.Fatalf("Failed to generate UUID %s: %s ", recordUUID, err)
+		} else if isValidUUID(recordUUID) == false {
+			t.Fatalf("Failed to validate UUID: %s ", recordUUID)
+		}
+	}
+}
+
+func Test_AppNames(t *testing.T) {
+	goodApps := []string{"penn", "teller", "a123"}
+	for _, value := range goodApps {
+		if isValidApp(value) == false {
+			t.Fatalf("Failed to validate good app name: %s ", value)
+		}
+	}
+	badApps := []string{"P1", "1as", "_a", "a_a", "a.a", "a a"}
+	for _, value := range badApps {
+		if isValidApp(value) == true {
+			t.Fatalf("Failed to validate bad app name: %s ", value)
+		}
+	}
+
+}
+
+func Test_getJSONPost(t *testing.T) {
+	goodJsons := []string{
+		`{"login":"abc","name": "tom", "pass": "mylittlepony", "admin": true}`,
+		`{"login":"1234","name": "tom", "pass": "mylittlepony", "admin": true}`,
+	}
+	for _, value := range goodJsons {
+		request := httptest.NewRequest("POST", "/user", strings.NewReader(value))
+		request.Header.Set("Content-Type", "application/json")
+		result, err := getJSONPost(request, "IL")
+		if err != nil {
+			t.Fatalf("Failed to parse json: %s, err: %s\n", value, err)
+		}
+		if len(result.loginIdx) == 0 {
+			t.Fatalf("Failed to parse login index from json: %s ", value)
+		}
+	}
+
+	badJsons := []string{
+		`{"login":true,"name": "tom", "pass": "mylittlepony", "admin": true}`,
+		`{"login":1,"name": "tom", "pass": "mylittlepony", "admin": true}`,
+		`{"login":null,"name": "tom", "pass": "mylittlepony", "admin": true}`,
+	}
+	for _, value := range badJsons {
+		request := httptest.NewRequest("POST", "/user", strings.NewReader(value))
+		request.Header.Set("Content-Type", "application/json")
+		result, err := getJSONPost(request, "IL")
+		if err != nil {
+			t.Fatalf("Failed to parse json: %s, err: %s\n", value, err)
+		}
+		if len(result.loginIdx) != 0 {
+			t.Fatalf("Failed to parse login index from json: %s ", value)
+		}
+	}
+}

src/xtokens_api.go

@@ -0,0 +1,148 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"reflect"
+	"strings"
+
+	"github.com/julienschmidt/httprouter"
+	"github.com/tidwall/gjson"
+)
+
+func (e mainEnv) userNewToken(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	userTOKEN := ps.ByName("token")
+	event := audit("create user temp access xtoken", userTOKEN)
+	defer func() { event.submit(e.db) }()
+
+	if enforceUUID(w, userTOKEN, event) == false {
+		return
+	}
+	if e.enforceAuth(w, r, event) == false {
+		return
+	}
+	records, err := getJSONPostData(r)
+	if err != nil {
+		returnError(w, r, "internal error", 405, err, event)
+		return
+	}
+	fields := ""
+	expiration := ""
+	appName := ""
+	if value, ok := records["fields"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			fields = value.(string)
+		}
+	}
+	if value, ok := records["expiration"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			expiration = value.(string)
+		} else {
+			returnError(w, r, "failed to parse expiration field", 405, err, event)
+			return
+		}
+	}
+	if value, ok := records["app"]; ok {
+		if reflect.TypeOf(value) == reflect.TypeOf("string") {
+			appName = strings.ToLower(value.(string))
+			if len(appName) > 0 && isValidApp(appName) == false {
+				returnError(w, r, "failed to parse app field", 405, nil, event)
+			}
+		} else {
+			// type is different
+			returnError(w, r, "failed to parse app field", 405, nil, event)
+		}
+	}
+	if len(expiration) == 0 {
+		returnError(w, r, "missing expiration field", 405, err, event)
+		return
+	}
+	xtokenUUID, err := e.db.generateUserTempXToken(userTOKEN, fields, expiration, appName)
+	if err != nil {
+		fmt.Println(err)
+		returnError(w, r, err.Error(), 405, err, event)
+		return
+	}
+	event.Msg = "Generated " + xtokenUUID
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	fmt.Fprintf(w, `{"status":"ok","xtoken":%q}`, xtokenUUID)
+}
+
+func (e mainEnv) userCheckToken(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
+	event := audit("get record by user temp access token", "")
+	defer func() { event.submit(e.db) }()
+
+	xtoken := ps.ByName("xtoken")
+	if enforceUUID(w, xtoken, event) == false {
+		return
+	}
+	authResult, err := e.db.checkUserAuthXToken(xtoken)
+	if err != nil {
+		fmt.Printf("%d access denied for : %s\n", http.StatusForbidden, xtoken)
+		w.WriteHeader(http.StatusForbidden)
+		w.Write([]byte("Access denied"))
+		return
+	}
+	var resultJSON []byte
+	if len(authResult.token) > 0 {
+		event.Record = authResult.token
+		event.App = authResult.appName
+		fmt.Printf("displaying fields: %s, user token: %s\n", authResult.fields, authResult.token)
+
+		if len(authResult.appName) > 0 {
+			resultJSON, err = e.db.getUserApp(authResult.token, authResult.appName)
+		} else {
+			resultJSON, err = e.db.getUser(authResult.token)
+		}
+		if err != nil {
+			returnError(w, r, "internal error", 405, err, event)
+			return
+		}
+		if resultJSON == nil {
+			returnError(w, r, "not found", 405, err, event)
+			return
+		}
+		fmt.Printf("Full user json: %s\n", resultJSON)
+		if len(authResult.fields) > 0 {
+			raw := make(map[string]interface{})
+			//var newJSON json
+			allFields := parseFields(authResult.fields)
+			for _, f := range allFields {
+				if f == "token" {
+					raw["token"] = authResult.token
+				} else {
+					value := gjson.Get(string(resultJSON), f)
+					//fmt.Printf("result %s -> %s\n", f, value)
+					/*
+						var raw2 map[string]interface{}
+						err = json.Unmarshal([]byte(value.String()), &raw2)
+						if err != nil {
+							fmt.Printf("Err: %s\n", err)
+						}
+					*/
+					raw[f] = value.Value()
+				}
+			}
+			resultJSON, _ = json.Marshal(raw)
+		}
+	}
+	//fmt.Fprintf(w, "<html><head><title>title</title></head>")
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(200)
+	var str string
+	if len(resultJSON) == 0 {
+		str = fmt.Sprintf(`{"status":"ok","type":"%s"}`, authResult.ttype)
+	} else {
+		if len(authResult.appName) > 0 {
+			str = fmt.Sprintf(`{"status":"ok","type":"%s","app":"%s","data":%s}`,
+				authResult.ttype, authResult.appName, resultJSON)
+		} else {
+			str = fmt.Sprintf(`{"status":"ok","type":"%s","data":%s}`,
+				authResult.ttype, resultJSON)
+		}
+	}
+	fmt.Printf("result: %s\n", str)
+	w.Write([]byte(str))
+}

src/xtokens_db.go

@@ -0,0 +1,174 @@
+package main
+
+import (
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	uuid "github.com/hashicorp/go-uuid"
+	"go.mongodb.org/mongo-driver/bson"
+)
+
+func (dbobj dbcon) getRootToken() (string, error) {
+	record, err := dbobj.getRecord(TblName.Xtokens, "type", "root")
+	if err != nil {
+		return "", err
+	}
+	if record == nil {
+		return "", nil
+	}
+	return record["xtoken"].(string), nil
+}
+
+func (dbobj dbcon) createRootToken() (string, error) {
+	rootToken, err := dbobj.getRootToken()
+	if len(rootToken) > 0 {
+		return rootToken, nil
+	}
+	rootToken, err = uuid.GenerateUUID()
+	if err != nil {
+		return "", err
+	}
+	bdoc := bson.M{}
+	bdoc["xtoken"] = rootToken
+	bdoc["type"] = "root"
+	_, err = dbobj.createRecord(TblName.Xtokens, bdoc)
+	if err != nil {
+		return rootToken, err
+	}
+	return rootToken, nil
+}
+
+func (dbobj dbcon) generateUserTempXToken(userTOKEN string, fields string, expiration string, appName string) (string, error) {
+	if isValidUUID(userTOKEN) == false {
+		return "", errors.New("bad uuid")
+	}
+	if len(expiration) == 0 {
+		return "", errors.New("failed to parse expiration")
+	}
+	if len(appName) > 0 {
+		apps, _ := dbobj.listAllApps()
+		if strings.Contains(string(apps), appName) == false {
+			return "", errors.New("app not found")
+		}
+	}
+
+	start, err := parseExpiration(expiration)
+	if err != nil {
+		return "", err
+	}
+
+	// check if user record exists
+	record, err := dbobj.lookupUserRecord(userTOKEN)
+	if record == nil || err != nil {
+		// not found
+		return "", errors.New("not found")
+	}
+
+	tokenUUID, err := uuid.GenerateUUID()
+	if err != nil {
+		return "", err
+	}
+	bdoc := bson.M{}
+	bdoc["token"] = userTOKEN
+	bdoc["xtoken"] = tokenUUID
+	bdoc["type"] = "temp"
+	bdoc["fields"] = fields
+	bdoc["endtime"] = start
+	if len(appName) > 0 {
+		bdoc["app"] = appName
+	}
+	_, err = dbobj.createRecord(TblName.Xtokens, bdoc)
+	if err != nil {
+		return "", err
+	}
+	return tokenUUID, nil
+}
+
+func (dbobj dbcon) generateUserLoginXToken(userTOKEN string) (string, error) {
+	if isValidUUID(userTOKEN) == false {
+		return "", errors.New("bad token format")
+	}
+
+	// check if user record exists
+	record, err := dbobj.lookupUserRecord(userTOKEN)
+	if record == nil || err != nil {
+		// not found
+		return "", errors.New("not found")
+	}
+
+	tokenUUID, err := uuid.GenerateUUID()
+	if err != nil {
+		return "", err
+	}
+	// by default login token for 30 minutes only
+	expired := int32(time.Now().Unix()) + 10*60
+	bdoc := bson.M{}
+	bdoc["token"] = userTOKEN
+	bdoc["xtoken"] = tokenUUID
+	bdoc["type"] = "login"
+	bdoc["endtime"] = expired
+	_, err = dbobj.createRecord(TblName.Xtokens, bdoc)
+	if err != nil {
+		return "", err
+	}
+	return tokenUUID, nil
+}
+
+func (dbobj dbcon) checkToken(tokenUUID string) bool {
+	//fmt.Printf("Token0 %s\n", tokenUUID)
+	if isValidUUID(tokenUUID) == false {
+		return false
+	}
+	record, err := dbobj.getRecord(TblName.Xtokens, "xtoken", tokenUUID)
+	if record == nil || err != nil {
+		return false
+	}
+	tokenType := record["type"].(string)
+	if tokenType == "root" {
+		return true
+	}
+	return false
+}
+
+func (dbobj dbcon) checkUserAuthXToken(xtokenUUID string) (tokenAuthResult, error) {
+	var result tokenAuthResult
+	if isValidUUID(xtokenUUID) == false {
+		return result, errors.New("failed to authenticate")
+	}
+	record, err := dbobj.getRecord(TblName.Xtokens, "xtoken", xtokenUUID)
+	if record == nil || err != nil {
+		return result, errors.New("failed to authenticate")
+	}
+	tokenType := record["type"].(string)
+	fmt.Printf("token type: %s\n", tokenType)
+	if tokenType == "root" {
+		// we have this admin user
+		result.ttype = "root"
+		result.name = "root"
+		return result, nil
+	}
+	result.name = xtokenUUID
+	// tokenType = temp
+	now := int32(time.Now().Unix())
+	if now > record["endtime"].(int32) {
+		return result, errors.New("token expired")
+	}
+	result.token = record["token"].(string)
+	if value, ok := record["fields"]; ok {
+		result.fields = value.(string)
+	}
+	if tokenType == "login" {
+		result.ttype = "login"
+	} else {
+		if value, ok := record["app"]; ok {
+			result.ttype = "app"
+			result.appName = value.(string)
+		} else {
+			result.ttype = "user"
+		}
+	}
+
+	return result, nil
+}

ui/index.html

@@ -0,0 +1,218 @@
+<!doctype html>
+<html>
+
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+    <title>Data Bunker Login</title>
+    <link href='http://fonts.googleapis.com/css?family=Roboto' rel='stylesheet' type='text/css'>
+    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
+        integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
+    <link rel="stylesheet" href="site/style.css">
+</head>
+
+<body>
+    <div class="container col-md-6 pY-120">
+        <div class="row col-12 col-md-12 ">
+            <div class="bigblock">
+                <h4>Login form</h4>
+                <p>Select login method and enter login details:</p>
+                <form id="loginform">
+                    <div class="form-group">
+                        <select onchange="changemethod(this);" class="custom-select" required id="keymethod">
+                            <option value="token">Token</option>
+                            <option value="Email">Email</option>
+                            <option value="Phone">Phone</option>
+                        </select>
+                    </div>
+                    <div class="form-group" id="email-conf-form" style="display:none;">
+                        <p>We will send you email with access code using <a target="_blank"
+                                href="https://www.mailgun.com/">https://www.mailgun.com/</a>&nbsp;-&nbsp;<a
+                                target="_blank" href="https://www.mailgun.com/gdpr/">Mailgun GDPR page</a></p>
+                        <div class="form-check">
+                            <input type="checkbox" class="form-check-input" onclick="hidealert();" id="emailchk">
+                            <label class="form-check-label" for="emailchk">Confirm to allow sending access code with
+                                mailgun.com</label>
+                        </div>
+                    </div>
+                    <div class="form-group" id="sms-conf-form" style="display:none;">
+                        <p>We will send you SMS with access code using <a target="_blank"
+                                href="https://www.twilio.com/">https://www.twilio.com/</a>&nbsp;-&nbsp;<a
+                                target="_blank" href="https://www.twilio.com/gdpr">Twilio GDPR page</a></p>
+                        <div class="form-check">
+                            <input type="checkbox" class="form-check-input" onclick="hidealert();" id="smschk">
+                            <label class="form-check-label" for="smschk">Confirm to allow sending access code with
+                                twilio.com</label>
+                        </div>
+                    </div>
+                    <div id="confalert" class="alert alert-warning" role="alert" style="display:none;">
+                        We can not send you access code!
+                    </div>
+                    <div id="badformat" class="alert alert-warning" role="alert" style="display:none;">
+                        Bad input value!
+                    </div>
+                    <div class="form-group">
+                        <input id="loginvalue" type="login" class="form-control" onclick="hidebadformat();"
+                            placeholder="Enter token...">
+                    </div>
+                    <div class="form-group">
+                        <div class="peers ai-c jc-sb fxw-nw">
+                            <div class="peer">
+                                <!--
+                                <div class="checkbox checkbox-circle checkbox-info peers ai-c">
+                                    <input type="checkbox" id="inputCall1" name="inputCheckboxesCall" class="peer">
+                                    <label for="inputCall1" class="peers peer-greed js-sb ai-c">
+                                        <span class="peer peer-greed">Remember Me</span></label></div>
+                                    -->
+                            </div>
+                            <div class="peer"><button onclick="return submitbtn();"
+                                    class="btn btn-primary">Login</button></div>
+                        </div>
+                    </div>
+                </form>
+            </div>
+        </div>
+    </div>
+    <script>
+        function isUUID(uuid) {
+            let s = "" + uuid;
+            s = s.match('^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$');
+            if (s === null) {
+                return false;
+            }
+            return true;
+        }
+        function hidealert() {
+            var confalert = document.getElementById('confalert');
+            confalert.style.display = "none";
+            var badformat = document.getElementById('badformat');
+            badformat.style.display = "none";
+        }
+        function hidebadformat() {
+            var badformat = document.getElementById('badformat');
+            badformat.style.display = "none";
+        }
+        function changemethod(obj) {
+            var element = document.getElementById('loginvalue');
+            var smsform = document.getElementById('sms-conf-form');
+            var emailform = document.getElementById('email-conf-form');
+            var smschk = document.getElementById('smschk');
+            var emailchk = document.getElementById('emailchk');
+            var confalert = document.getElementById('confalert');
+            var badformat = document.getElementById('badformat');
+
+            if (!element || !smsform || !emailform ||
+                !smschk || !emailchk || !confalert || !badformat) {
+                return false;
+            }
+            var value = obj.value;
+            if (!value) {
+                return false;
+            }
+            badformat.style.display = "none";
+            smschk.checked = false;
+            emailchk.checked = false;
+            value = value.toLowerCase();
+            var key = element.placeholder = "Enter " + value + "...";
+            confalert.style.display = "none";
+            if (value == "email") {
+                smsform.style.display = "none";
+                emailform.style.display = "block";
+            } else if (value == "phone") {
+                smsform.style.display = "block";
+                emailform.style.display = "none";
+            } else {
+                smsform.style.display = "none";
+                emailform.style.display = "none";
+            }
+        }
+
+        function submitbtn() {
+            var element = document.getElementById('loginvalue')
+            var smschk = document.getElementById('smschk');
+            var emailchk = document.getElementById('emailchk');
+            var confalert = document.getElementById('confalert');
+            var keymethod = document.getElementById('keymethod');
+            var badformat = document.getElementById('badformat');
+
+            if (!element || !smschk || !emailchk || !confalert || !keymethod) {
+                return false;
+            }
+            var key = element.value;
+            if (!key) {
+                return false;
+            }
+            var kkk = keymethod.options[keymethod.selectedIndex].value;
+            if ((kkk == "Email" && emailchk.checked == false) ||
+                (kkk == "Phone" && smschk.checked == false)) {
+                confalert.style.display = "block";
+                return false;
+            }
+
+
+            if (kkk == "Token" && isUUID(key) == true) {
+                var xhr = new XMLHttpRequest();
+                xhr.open('GET', "/v1/xtoken/" + key);
+                xhr.onload = function () {
+                    if (xhr.status === 200) {
+                        var data = JSON.parse(xhr.responseText);
+                        if (data && data.status && data.status == "ok") {
+                            window.localStorage.setItem('xtoken', key);
+                            window.localStorage.setItem('type', data.type);
+                            if (data.data) {
+                                document.location = "/site/display-data.html";
+                            } else {
+                                document.location = "/site/admin-events.html";
+                            }
+                        }
+                    }
+                };
+                xhr.send();
+            } else if (kkk == "Email" && key.indexOf('@') > 0) {
+                var params = 'brief=send-email-mailgun-on-login';
+                window.localStorage.setItem('login', key);
+                var xhr0 = new XMLHttpRequest();
+                // first save consent
+                xhr0.open('POST', "/v1/consent/email/" + encodeURI(key));
+                xhr0.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
+                xhr0.onload = function () {
+                    if (xhr0.status === 200) {
+                        var xhr = new XMLHttpRequest();
+                        xhr.open('GET', "/v1/login/email/" + encodeURI(key));
+                        xhr.onload = function () {
+                            if (xhr.status === 200) {
+                                document.location = "/site/enter.html";
+                            }
+                        }
+                        xhr.send();
+                    }
+                }
+                xhr0.send(params);
+
+
+            } else if (kkk == "Phone") {
+                window.localStorage.setItem('login', key);
+                var params = 'brief=send-sms-twilio-on-login';
+                var xhr0 = new XMLHttpRequest();
+                xhr0.open('POST', "/v1/consent/phone/" + encodeURI(key));
+                xhr0.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
+                xhr0.onload = function () {
+                    if (xhr0.status === 200) {
+                        var xhr = new XMLHttpRequest();
+                        xhr.open('GET', "/v1/login/phone/" + encodeURI(key));
+                        xhr.onload = function () {
+                            if (xhr.status === 200) {
+                                document.location = "/site/enter.html";
+                            }
+                        }
+                        xhr.send();
+                    }
+                }
+                xhr0.send(params);
+            } else {
+                badformat.style.display = "block";
+            }
+            return false;
+        }
+    </script>
+</body>

ui/site/display-data.html

@@ -0,0 +1,57 @@
+<!doctype html>
+<html>
+
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+    <title>Data Bunker Login</title>
+    <link href='http://fonts.googleapis.com/css?family=Roboto' rel='stylesheet' type='text/css'>
+    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
+        integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.6.3/css/all.css"
+        integrity="sha384-UHRtZLI+pbxtHCWp1t77Bi1L4ZtiqrqD80Kn4Z8NTSRyMA2Fd33n5dQ8lWUE00s/" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://unpkg.com/bootstrap-table@1.15.5/dist/bootstrap-table.min.css">
+    <link rel="stylesheet" href="style.css">
+
+    <script src="https://code.jquery.com/jquery-3.3.1.min.js"
+        integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"
+        integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1"
+        crossorigin="anonymous"></script>
+    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"
+        integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM"
+        crossorigin="anonymous"></script>
+
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/highlight.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/languages/json.min.js"></script>
+    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/styles/a11y-dark.min.css">
+</head>
+
+<body>
+    <div class="container col-md-6 pY-120">
+        <div class="row col-12 col-md-12 ">
+            <div class="bigblock">
+                <h4>Record display</h4>
+                <p id="msg">text</p>
+                <pre id="data"></pre>
+            </div>
+        </div>
+    </div>
+    <script>
+        var xtoken = window.localStorage.getItem('xtoken');
+        var ttype = window.localStorage.getItem('type');
+        $('#msg').val("Display: " + ttype + " " + xtoken);
+        $.get("/v1/xtoken/" + xtoken, function (data) {
+            if (data.status == "ok") {
+                $('#msg').text("Access xtoken value: "+xtoken)
+                var d = JSON.stringify(data.data, null, 4);
+                $('#data').append('<code class="json">' + d + '</code>');
+                document.querySelectorAll('pre code').forEach((block) => {
+                    hljs.highlightBlock(block);
+                });
+            } else {
+                alert("error");
+            }
+        }, "json");
+    </script>
+</body>

ui/site/enter.html

@@ -0,0 +1,82 @@
+<!doctype html>
+<html>
+
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+    <title>Data Bunker Login</title>
+    <link href='http://fonts.googleapis.com/css?family=Roboto' rel='stylesheet' type='text/css'>
+    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
+        integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.6.3/css/all.css"
+        integrity="sha384-UHRtZLI+pbxtHCWp1t77Bi1L4ZtiqrqD80Kn4Z8NTSRyMA2Fd33n5dQ8lWUE00s/" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://unpkg.com/bootstrap-table@1.15.5/dist/bootstrap-table.min.css">
+    <link rel="stylesheet" href="style.css">
+
+    <script src="https://code.jquery.com/jquery-3.3.1.min.js"
+        integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"
+        integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1"
+        crossorigin="anonymous"></script>
+    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"
+        integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM"
+        crossorigin="anonymous"></script>
+</head>
+
+<body>
+    <div class="container col-md-6 pY-120">
+        <div class="row col-12 col-md-12 ">
+            <div class="bigblock">
+                <h4>Verification step</h4>
+                <p>Enter the code you received by email or SMS</p>
+                <form id="loginform">
+                    <div class="form-group"><label class="text-normal text-dark">Enter code</label>
+                        <input id="codevalue" type="login" class="form-control" placeholder="Enter..."></div>
+                    <div class="form-group">
+                        <div class="peers ai-c jc-sb fxw-nw">
+                            <div class="peer">
+                                <!--
+                                <div class="checkbox checkbox-circle checkbox-info peers ai-c">
+                                    <input type="checkbox" id="inputCall1" name="inputCheckboxesCall" class="peer">
+                                    <label for="inputCall1" class="peers peer-greed js-sb ai-c">
+                                        <span class="peer peer-greed">Remember Me</span></label></div>
+                                    -->
+                            </div>
+                            <div class="peer"><button id="submitbtn" class="btn btn-primary">Enter</button></div>
+                        </div>
+                    </div>
+                </form>
+            </div>
+        </div>
+    </div>
+    <script>
+        $('#submitbtn').on('click', function (e) {
+            e.preventDefault();
+            var code = $('#codevalue').val();
+            var login = window.localStorage.getItem('login')
+            if (login.indexOf('@') > 0) {
+                $.get("/v1/enter/email/" + encodeURI(login) + "/" + encodeURI(code), function (data) {
+                    window.localStorage.setItem('login', "");
+                    if (data.xtoken) {
+                        window.localStorage.setItem('xtoken', data.xtoken);
+                        window.localStorage.setItem('token', data.token);
+                        document.location = "user-profile.html";
+                    } else {
+                        document.location = "/";
+                    }
+                }, "json");
+            } else {
+                $.get("/v1/enter/phone/" + encodeURI(login) + "/" + encodeURI(code), function (data) {
+                    window.localStorage.setItem('login', "");
+                    if (data.xtoken) {
+                        window.localStorage.setItem('xtoken', data.xtoken);
+                        window.localStorage.setItem('token', data.token);
+                        document.location = "user-profile.html";
+                    } else {
+                        document.location = "/";
+                    }
+                }, "json");
+            }
+        })
+    </script>
+</body>

ui/site/index.html

@@ -0,0 +1,8 @@
+<html>
+<head>
+<title></title>
+</head>
+<body>
+    <script>document.location = "../"; </script>
+</body>
+</html>

ui/site/site.js

@@ -0,0 +1,16 @@
+function bunker_logout()
+{
+    localStorage.removeItem("xtoken");
+    localStorage.removeItem("xtoken");
+    localStorage.removeItem("login");
+    document.location = "/";
+}
+
+function dateFormat(value, row, index) {
+    //return moment(value).format('DD/MM/YYYY');
+    var d = new Date(parseInt(value) * 1000);
+    let f_date = d.getFullYear() + "-" + (d.getMonth() + 1) + "-" + d.getDate() +
+      " " + d.getHours() + ":" + d.getMinutes() + ":" + d.getSeconds()
+    //return d.toUTCString();
+    return f_date;
+}

ui/site/style.css

@@ -0,0 +1,113 @@
+html {
+    height: 100%;
+    -webkit-font-smoothing: antialiased;
+}
+body, h1, h2, h3, h4, h5, h6 {
+    font-family: Roboto,-apple-system,system-ui,BlinkMacSystemFont,Segoe UI,Oxygen,Ubuntu,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,Arial,sans-serif;
+    line-height: 1.5;
+}
+body {
+  font-size: 14px;
+  color: #72777a;
+  letter-spacing: 0.2px;
+  font-weight: 400;
+  padding:0;
+  margin:0;
+  height: 100%;
+  background-color: #f9fafb !important;
+}
+.container {
+    height: 100%;
+    width: 100%;
+    display: block;
+}
+.bigblock {
+  background-color: #fff !important;
+  border-radius: 3px !important;
+  border: 1px solid rgba(0, 0, 0, 0.0625) !important;
+  padding: 20px !important;
+  width: 100%;
+}
+div.bigblock table {
+    color: #72777a;
+}
+h4 {
+    color: #313435!important;
+    margin-bottom: 20px!important;
+    letter-spacing: .5px;
+    font-size: 1.3125rem;
+}
+.peer {
+    display: block;
+    height: auto;
+    -webkit-box-flex: 0;
+    -ms-flex: 0 0 auto;
+    flex: 0 0 auto;
+}
+.peers {
+    -webkit-box-sizing: border-box;
+    box-sizing: border-box;
+    display: -webkit-box!important;
+    display: -ms-flexbox!important;
+    display: flex!important;
+    -webkit-box-align: start;
+    -ms-flex-align: start;
+    align-items: flex-start;
+    -webkit-box-pack: start;
+    -ms-flex-pack: start;
+    justify-content: flex-start;
+    -webkit-box-orient: horizontal;
+    -webkit-box-direction: normal;
+    -ms-flex-flow: row wrap;
+    flex-flow: row wrap;
+    height: auto;
+    max-width: 100%;
+    margin: 0;
+    padding: 0;
+}
+.jc-sb {
+    -webkit-box-pack: justify;
+    -ms-flex-pack: justify;
+    justify-content: space-between;
+}
+.ai-c {
+    -webkit-box-align: center;
+    -ms-flex-align: center;
+    align-items: center;
+}
+.fxw-nw {
+    -ms-flex-wrap: nowrap;
+    flex-wrap: nowrap;
+}
+.btn-primary {
+    color: #fff;
+    background-color: #2196f3;
+    border-color: #2196f3;
+}
+.btn {
+    display: inline-block;
+    font-weight: 400;
+    text-align: center;
+    white-space: nowrap;
+    vertical-align: middle;
+    -webkit-user-select: none;
+    -moz-user-select: none;
+    -ms-user-select: none;
+    user-select: none;
+    border: 1px solid transparent;
+    padding: 6px 12px;
+    padding: .375rem .75rem;
+    font-size: 14px;
+    font-size: .875rem;
+    line-height: 1.5;
+    border-radius: .25rem;
+    -webkit-transition: background-color .15s ease-in-out,border-color .15s ease-in-out,-webkit-box-shadow .15s ease-in-out;
+    transition: background-color .15s ease-in-out,border-color .15s ease-in-out,-webkit-box-shadow .15s ease-in-out;
+    -o-transition: background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out;
+    transition: background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out;
+    transition: background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out,-webkit-box-shadow .15s ease-in-out;
+}
+.pY-120 {
+    padding-top: 120px!important;
+    padding-bottom: 120px!important;
+}

ui/site/user-apps.html

@@ -0,0 +1,110 @@
+<!doctype html>
+<html>
+
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+    <title>Data Bunker - list of apps</title>
+    <link href='http://fonts.googleapis.com/css?family=Roboto' rel='stylesheet' type='text/css'>
+    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
+        integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.6.3/css/all.css"
+        integrity="sha384-UHRtZLI+pbxtHCWp1t77Bi1L4ZtiqrqD80Kn4Z8NTSRyMA2Fd33n5dQ8lWUE00s/" crossorigin="anonymous">
+     <link rel="stylesheet" href="style.css">
+
+    <script src="https://code.jquery.com/jquery-3.3.1.min.js"
+        integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"
+        integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1"
+        crossorigin="anonymous"></script>
+    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"
+        integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM"
+        crossorigin="anonymous"></script>
+
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/highlight.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/languages/json.min.js"></script>
+    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/styles/a11y-dark.min.css">
+    <script src="site.js"></script>
+</head>
+
+<body>
+    <div class="container col-md-6">
+        <div class="row col-12 col-md-12 ">
+            <div style="width:100%;">
+                <nav class="navbar navbar-expand-lg navbar-light bg-light">
+                    <a class="navbar-brand" href="#">Menu</a>
+                    <button class="navbar-toggler" type="button" data-toggle="collapse"
+                        data-target="#navbarNavAltMarkup" aria-controls="navbarNavAltMarkup" aria-expanded="false"
+                        aria-label="Toggle navigation">
+                        <span class="navbar-toggler-icon"></span>
+                    </button>
+                    <div class="collapse navbar-collapse" id="navbarNavAltMarkup">
+                        <div class="navbar-nav">
+                            <a class="nav-item nav-link" href="user-profile.html">Profile</a>
+                            <a class="nav-item nav-link active" href="user-apps.html">Apps</a>
+                            <a class="nav-item nav-link" href="user-data-processing.html">Confirmations</a>
+                            <a class="nav-item nav-link" href="user-audit.html">Audit</a>
+                            <a class="nav-item nav-link" href="javascript:bunker_logout();">Logout</a>
+                        </div>
+                    </div>
+                </nav>
+            </div>
+            <div class="bigblock">
+                <h4>User apps data</h4>
+                <p id="msg">text</p>
+                <pre id="data"></pre>
+            </div>
+        </div>
+    </div>
+    <script>
+        function fetchApp(token, xtoken, app) {
+            var xhr0 = new XMLHttpRequest();
+            // first save consent
+            xhr0.open('GET', '/v1/userapp/token/' + token + "/" + app);
+            xhr0.setRequestHeader("X-Bunker-Token", xtoken)
+            xhr0.setRequestHeader('Content-type', 'application/json');
+            xhr0.onload = function () {
+                if (xhr0.status === 200) {
+                    var data = JSON.parse(xhr0.responseText);
+                    if (data.status == "ok") {
+                        var d = JSON.stringify(data.data, null, 4);
+                        $('#data').append('<h4>'+app+'</h4><code class="json">' + d + '</code>');
+                        document.querySelectorAll('pre code').forEach((block) => {
+                            hljs.highlightBlock(block);
+                        });
+                    }
+                //} else if (xhr0.status > 400 && xhr0.status < 500) {
+                //    document.location = "/";
+                }
+            }
+            xhr0.send();
+
+        }
+        var xtoken = window.localStorage.getItem('xtoken');
+        var token = window.localStorage.getItem('token');
+        var ttype = window.localStorage.getItem('type');
+        $('#msg').val("Display: " + ttype + " " + token);
+
+        var xhr = new XMLHttpRequest();
+        xhr.open('GET', '/v1/userapp/token/' + token);
+        xhr.setRequestHeader("X-Bunker-Token", xtoken)
+        xhr.setRequestHeader('Content-type', 'application/json');
+        xhr.onload = function () {
+            if (xhr.status === 200) {
+                var data = JSON.parse(xhr.responseText);
+                if (data.apps) {
+                    var index;
+                    for (index = 0; index < data.apps.length; ++index) {
+                        var app = data.apps[index];
+                        console.log("app", app)
+                        fetchApp(token, xtoken, app);
+                    }
+                }
+            } else if (xhr.status > 400 && xhr.status < 500) {
+                document.location = "/";
+            }
+        }
+        xhr.send()
+
+    </script>
+</body>

ui/site/user-audit.html

@@ -0,0 +1,102 @@
+<!doctype html>
+<html>
+
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+  <title>Data Bunker - audit events</title>
+  <link href='http://fonts.googleapis.com/css?family=Roboto' rel='stylesheet' type='text/css'>
+  <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
+    integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
+  <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.6.3/css/all.css"
+    integrity="sha384-UHRtZLI+pbxtHCWp1t77Bi1L4ZtiqrqD80Kn4Z8NTSRyMA2Fd33n5dQ8lWUE00s/" crossorigin="anonymous">
+  <link rel="stylesheet" href="https://unpkg.com/bootstrap-table@1.15.5/dist/bootstrap-table.min.css">
+  <link rel="stylesheet" href="style.css">
+
+  <script src="https://code.jquery.com/jquery-3.3.1.min.js"
+    integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
+  <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"
+    integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1"
+    crossorigin="anonymous"></script>
+  <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"
+    integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM"
+    crossorigin="anonymous"></script>
+  <script src="https://unpkg.com/bootstrap-table@1.15.5/dist/bootstrap-table.min.js"></script>
+  <script src="https://unpkg.com/bootstrap-table@1.15.5/dist/bootstrap-table.min.js"></script>
+
+  <script>
+    $(function () {
+      var xtoken = window.localStorage.getItem('xtoken');
+      var token = window.localStorage.getItem('token');
+      var ttype = window.localStorage.getItem('type');
+      $('#msg').text("Access xtoken value: " + xtoken + " user: " + token)
+      //token = "faa006da-475e-45c6-a4a1-6586dce8b8d2";
+      $('#table').bootstrapTable({
+        /*data: mydata */
+        url: "http://localhost:3000/v1/audit/list/" + token,
+        undefinedText: 'n/a',
+        /* url: "data1.json", */
+        method: "GET",
+        ajaxOptions: {
+          headers: { "X-Bunker-Token": xtoken },
+          crossDomain: true
+        },
+        showExtendedPagination: true,
+        sidePagination: "server",
+        pagination: true,
+        search: false,
+        classes: "table",
+        onLoadError: function (status, res) {
+          console.log(status);
+          if (status > 400 && status < 500) {
+            document.location = "/";
+          }
+        }
+      });
+    });
+  </script>
+  <script src="site.js"></script>
+</head>
+
+<body>
+  <div class="container">
+    <div class="row col-md-12">
+      <div style="width:100%;">
+        <nav class="navbar navbar-expand-lg navbar-light bg-light">
+          <a class="navbar-brand" href="#">Menu</a>
+          <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarNavAltMarkup"
+            aria-controls="navbarNavAltMarkup" aria-expanded="false" aria-label="Toggle navigation">
+            <span class="navbar-toggler-icon"></span>
+          </button>
+          <div class="collapse navbar-collapse" id="navbarNavAltMarkup">
+            <div class="navbar-nav">
+              <a class="nav-item nav-link" href="user-profile.html">Profile <span class="sr-only">(current)</span></a>
+              <a class="nav-item nav-link" href="user-apps.html">Apps</a>
+              <a class="nav-item nav-link" href="user-data-processing.html">Confirmations</a>
+              <a class="nav-item nav-link active" href="user-audit.html">Audit</a>
+              <a class="nav-item nav-link" href="javascript:bunker_logout();">Logout</a>
+            </div>
+          </div>
+        </nav>
+      </div>
+      <div class="bigblock">
+        <h4>Audit table</h4>
+        <p id="msg">text here, text, text</p>
+        <table id="table" class="table">
+          <thead>
+            <tr>
+              <th scope="col" data-field="when" data-formatter="dateFormat">when</th>
+              <th scope="col" data-field="who">Who</th>
+              <th scope="col" data-field="app">App</th>
+              <th scope="col" data-field="title">Title</th>
+              <th scope="col" data-field="status">Status</th>
+              <th scope="col" data-field="msg">Msg</th>
+            </tr>
+          </thead>
+        </table>
+      </div>
+    </div>
+  </div>
+</body>
+
+</html>

ui/site/user-data-processing.html

@@ -0,0 +1,166 @@
+<!doctype html>
+<html>
+
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+    <title>Data Bunker - List of data processing requests</title>
+    <link href='http://fonts.googleapis.com/css?family=Roboto' rel='stylesheet' type='text/css'>
+    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
+        integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.6.3/css/all.css"
+        integrity="sha384-UHRtZLI+pbxtHCWp1t77Bi1L4ZtiqrqD80Kn4Z8NTSRyMA2Fd33n5dQ8lWUE00s/" crossorigin="anonymous">
+    <link rel="stylesheet" href="style.css">
+
+    <script src="https://code.jquery.com/jquery-3.3.1.min.js"
+        integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"
+        integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1"
+        crossorigin="anonymous"></script>
+    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"
+        integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM"
+        crossorigin="anonymous"></script>
+    <script src="site.js"></script>
+</head>
+
+<body>
+    <div class="container">
+        <div class="row col-md-12">
+            <div style="width:100%;">
+                <nav class="navbar navbar-expand-lg navbar-light bg-light">
+                    <a class="navbar-brand" href="#">Menu</a>
+                    <button class="navbar-toggler" type="button" data-toggle="collapse"
+                        data-target="#navbarNavAltMarkup" aria-controls="navbarNavAltMarkup" aria-expanded="false"
+                        aria-label="Toggle navigation">
+                        <span class="navbar-toggler-icon"></span>
+                    </button>
+                    <div class="collapse navbar-collapse" id="navbarNavAltMarkup">
+                        <div class="navbar-nav">
+                            <a class="nav-item nav-link" href="user-profile.html">Profile <span
+                                    class="sr-only">(current)</span></a>
+                            <a class="nav-item nav-link" href="user-apps.html">Apps</a>
+                            <a class="nav-item nav-link active" href="user-data-processing.html">Confirmations</a>
+                            <a class="nav-item nav-link" href="user-audit.html">Audit</a>
+                            <a class="nav-item nav-link" href="javascript:bunker_logout();">Logout</a>
+                        </div>
+                    </div>
+                </nav>
+            </div>
+            <div class="bigblock">
+                <h4>Data processing requests</h4>
+                <p id="msg">List of all user consents. You can allow and cancel your consent request here.</p>
+                <div id="data"></div>
+            </div>
+        </div>
+    </div>
+    <script>
+        var xtoken = window.localStorage.getItem('xtoken');
+        var token = window.localStorage.getItem('token');
+
+        var xhr = new XMLHttpRequest();
+        xhr.open('GET', "http://localhost:3000/v1/consent/token/" + token);
+        xhr.setRequestHeader("X-Bunker-Token", xtoken)
+        xhr.setRequestHeader('Content-type', 'application/json');
+        xhr.onload = function () {
+            if (xhr.status === 200) {
+                var data = JSON.parse(xhr.responseText);
+                if (data.status == "ok") {
+                    //$('#msg').text("Access xtoken value: " + xtoken + " user: " + token)
+                    console.log(data)
+                    var index;
+                    for (index = 0; index < data.rows.length; ++index) {
+                        var row = data.rows[index];
+                        //console.log("row", row)
+                        $('#data').append(prepareRow(row));
+                        //fetchApp(token, xtoken, app);
+                    }
+                }
+            } else if (xhr.status > 400 && xhr.status < 500) {
+                document.location = "/";
+            }
+        }
+        xhr.send();
+        function cancelThis(brief) {
+            var heading = "Confirm action";
+            var question = "Are you sure?";
+            var cancelButtonTxt = "Close popup";
+            var okButtonTxt = "Cancel!";
+            var confirmModal =
+                $('<div class="modal fade" role="dialog"><div class="modal-dialog" role="document"><div class="modal-content">' +
+                    '<div class="modal-header">' +
+                    '<h5 class="modal-title">' + heading + '</h5>' +
+                    '<button type="button" class="close" data-dismiss="modal" aria-label="Close">' +
+                        '<span aria-hidden="true">&times;</span></button>' +
+                    '</div>' +
+                    '<div class="modal-body">' +
+                    '<p>' + question + '</p>' +
+                    '</div>' +
+                    '<div class="modal-footer">' +
+                    '<a href="#" class="btn" data-dismiss="modal">' +
+                    cancelButtonTxt +
+                    '</a>' +
+                    '<a href="#" id="okButton" class="btn btn-primary">' +
+                    okButtonTxt +
+                    '</a>' +
+                    '</div>' +
+                    '</div></div></div>');
+
+            confirmModal.find('#okButton').click(function (event) {
+                //callback();
+                cancelThisDo(brief);
+                confirmModal.modal('hide');
+            });
+
+            confirmModal.modal('show');
+        }
+        function cancelThisDo(brief) {
+            var params = 'brief=' + brief;
+            var xhr = new XMLHttpRequest();
+            xhr.open('DELETE', "/v1/consent/token/" + token);
+            xhr.setRequestHeader("X-Bunker-Token", xtoken)
+            xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
+            xhr.onload = function () {
+                if (xhr.status === 200) {
+                    document.location.reload();
+                } else if (xhr.status > 400 && xhr.status < 500) {
+                    document.location = "/";
+                }
+            }
+            xhr.send(params);
+        }
+        function acceptThis(brief) {
+            var params = 'brief=' + brief;
+            var xhr = new XMLHttpRequest();
+            xhr.open('POST', "/v1/consent/token/" + token);
+            xhr.setRequestHeader("X-Bunker-Token", xtoken)
+            xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
+            xhr.onload = function () {
+                if (xhr.status === 200) {
+                    document.location.reload();
+                } else if (xhr.status > 400 && xhr.status < 500) {
+                    document.location = "/";
+                }
+            }
+            xhr.send(params);
+        }
+        function prepareRow(row) {
+            var msg = row.brief;
+            if (row.message != "") {
+                msg = row.message;
+            }
+            var start = '<div class="d-flex flex-row bd-highlight mb-4">';
+            var d = '<div class="p-2 bd-highlight">When: ' + dateFormat(row.when) + '</div>'
+            var identity = '<div class="p-2 flex-fill bd-highlight">Identity: ' + row.who + ' (' + row.type + ')<br/>' + msg + '</div>'
+            var cancel = "<a href=\"javascript:cancelThis('" + row.brief + "');\">cancel</a>";
+            var accept = "<a href=\"javascript:acceptThis('" + row.brief + "');\">accept</a>";
+            var op = cancel;
+            if (row.status == 'cancel') {
+                op = accept;
+            }
+            var status = '<div class="p-2 bd-highlight">Current status: ' + row.status + '</br><center>' + op + '</center></div>'
+            return start + d + identity + status + '</div>';
+        }
+    </script>
+</body>
+
+</html>

ui/site/user-profile.html

@@ -0,0 +1,146 @@
+<!doctype html>
+<html>
+
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
+    <title>Data Bunker - user profile</title>
+    <link href='http://fonts.googleapis.com/css?family=Roboto' rel='stylesheet' type='text/css'>
+    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
+        integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.6.3/css/all.css"
+        integrity="sha384-UHRtZLI+pbxtHCWp1t77Bi1L4ZtiqrqD80Kn4Z8NTSRyMA2Fd33n5dQ8lWUE00s/" crossorigin="anonymous">
+    <link rel="stylesheet" href="https://unpkg.com/bootstrap-table@1.15.5/dist/bootstrap-table.min.css">
+    <link rel="stylesheet" href="style.css">
+
+    <script src="https://code.jquery.com/jquery-3.3.1.min.js"
+        integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"
+        integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1"
+        crossorigin="anonymous"></script>
+    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"
+        integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM"
+        crossorigin="anonymous"></script>
+
+    <link href="https://cdnjs.cloudflare.com/ajax/libs/jsoneditor/7.0.4/jsoneditor.min.css" rel="stylesheet"
+        type="text/css">
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/jsoneditor/7.0.4/jsoneditor.min.js"></script>
+    <script src="site.js"></script>
+</head>
+
+<body>
+    <div class="container col-md-6">
+        <div class="row col-12 col-md-12 ">
+            <div style="width:100%;">
+                <nav class="navbar navbar-expand-lg navbar-light bg-light">
+                    <a class="navbar-brand" href="#">Menu</a>
+                    <button class="navbar-toggler" type="button" data-toggle="collapse"
+                        data-target="#navbarNavAltMarkup" aria-controls="navbarNavAltMarkup" aria-expanded="false"
+                        aria-label="Toggle navigation">
+                        <span class="navbar-toggler-icon"></span>
+                    </button>
+                    <div class="collapse navbar-collapse" id="navbarNavAltMarkup">
+                        <div class="navbar-nav">
+                            <a class="nav-item nav-link active" href="#">Profile <span
+                                    class="sr-only">(current)</span></a>
+                            <a class="nav-item nav-link" href="user-apps.html">Apps</a>
+                            <a class="nav-item nav-link" href="user-data-processing.html">Confirmations</a>
+                            <a class="nav-item nav-link" href="user-audit.html">Audit</a>
+                            <a class="nav-item nav-link" href="javascript:bunker_logout();">Logout</a>
+                        </div>
+                    </div>
+                </nav>
+            </div>
+            <div class="bigblock">
+                <h4>User record profile</h4>
+                <p id="msg">text</p>
+                <div id="jsoneditor" style="width: 400px; height: 400px;"></div>
+                <pre id="data"></pre>
+
+                <div class="form-group">
+                    <div class="peers ai-c jc-sb fxw-nw">
+                        <div class="peer">
+                            <!--
+                            <div class="checkbox checkbox-circle checkbox-info peers ai-c">
+                                <input type="checkbox" id="inputCall1" name="inputCheckboxesCall" class="peer">
+                                <label for="inputCall1" class="peers peer-greed js-sb ai-c">
+                                    <span class="peer peer-greed">Remember Me</span></label></div>
+                                -->
+                        </div>
+                        <div class="peer"><button onclick="return savebtn();" class="btn btn-primary">Save</button>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+    <script>
+        var xtoken = window.localStorage.getItem('xtoken');
+        var token = window.localStorage.getItem('token');
+        var ttype = window.localStorage.getItem('type');
+        var pKeys = ["password", "pwd", "pass", "parol", "passwd"];
+        var editor;
+        function savebtn() {
+            const updatedJson = editor.get()
+            var params = JSON.stringify(updatedJson);
+            //alert(d);
+            var xhr0 = new XMLHttpRequest();
+            // first save consent
+            xhr0.open('PUT', "/v1/user/token/" + token);
+            xhr0.setRequestHeader("X-Bunker-Token", xtoken)
+            xhr0.setRequestHeader('Content-type', 'application/json');
+            xhr0.onload = function () {
+                //if (xhr0.status === 200) {
+                //    //
+                //} else 
+                if (xhr0.status > 400 && xhr0.status < 500) {
+                    document.location = "/";
+                }
+            }
+            xhr0.send(params);
+        }
+        $('#msg').val("Display: " + ttype + " " + token);
+        /*
+        $.get("/v1/token/" + token, function (data) {
+        }, "json");
+        */
+        var xhr = new XMLHttpRequest();
+        xhr.open('GET', '/v1/user/token/' + token);
+        xhr.setRequestHeader("X-Bunker-Token", xtoken)
+        xhr.setRequestHeader('Content-type', 'application/json');
+        xhr.onload = function () {
+            if (xhr.status === 200) {
+                var data = JSON.parse(xhr.responseText);
+                if (data.status == "ok") {
+                    $('#msg').text("Access xtoken value: " + xtoken + " user: " + token)
+                    // create the editor
+                    const container = document.getElementById("jsoneditor")
+                    const options = { "mode": "form" }
+                    editor = new JSONEditor(container, options)
+                    // remove password field from displaying to user
+                    for (var key in data.data) {
+                        var k0 = key.toLowerCase();
+                        if (pKeys.includes(k0)) {
+                            delete data.data[key];
+                        }
+                    }
+                    editor.set(data.data)
+
+                    // get json
+                    const updatedJson = editor.get()
+                    /*
+                    var d = JSON.stringify(data.data, null, 4);
+                    $('#data').append('<code class="json">' + d + '</code>');
+                    document.querySelectorAll('pre code').forEach((block) => {
+                        hljs.highlightBlock(block);
+                    });
+                    */
+                }
+            } else if (xhr.status > 400 && xhr.status < 500) {
+                document.location = "/";
+            }
+        }
+        xhr.send();
+    </script>
+
+</body>