This is a general link for logging into the billing system on behalf of
a server, but it's tied to the .contact_email and takes the user
straight to the /deactivate/ page via the next_page mechanism.
Given that most of the use cases for realms-only code path would
really like to upload audit logs too, and the others would likely
produce a better user experience if they upoaded audit logs, we
should just have a single main code path here i.e.
'send_analytics_to_push_bouncer'.
We still only upload usage statistics according to documented
option, and only from the analytics cron job.
The error handling takes place in 'send_analytics_to_push_bouncer'
itself.
I accidentally free trials for both cloud and self hosted
enabled while testing, hence didn't catch it.
This mostly involves fixing `is_free_trial_offer_enabled` to
return the correct value and providing it the correct input.
The way the flow goes now is this:
1. The user initiaties login via "Billing" in the gear menu.
2. That takes them to `/self-hosted-billing/` (possibly with a
`next_page` param if we use that for some gear menu options).
3. The server queries the bouncer to give the user a link with a signed
access token.
4. The user is redirected to that link (on `selfhosting.zulipchat.com`).
Now we have two cases, either the user is logging in for the first time
and already did in the past.
If this is the first time, we have:
5. The user is asked to fill in their email in a form that's shown,
pre-filled with the value provided inside the signed access token.
They POST this to the next endpoint.
6. The next endpoint sends a confirmation email to that address and asks
the user to go check their email.
7. The user clicks the link in their email is taken to the
from_confirmation endpoint.
8. Their initial RemoteBillingUser is created, a new signed link like in
(3) is generated and they're transparently taken back to (4),
where now that they have a RemoteBillingUser, they're handled
just like a user who already logged in before:
If the user already logged in before, they go straight here:
9. "Confirm login" page - they're shown their information (email and
full_name), can update
their full name in the form if they want. They also accept ToS here
if necessary. They POST this form back to
the endpoint and finally have a logged in session.
10. They're redirected to billing (or `next_page`) now that they have
access.
For the last form (with Full Name and ToS consent field), this pretty
shamelessly re-uses and directly renders the
corporate/remote_realm_billing_finalize_login_confirmation.html
template. That's probably good in terms of re-use, but calls for a
clean-up commit that will generalize the name of this template and the
classes/ids in the HTML.
If the RemoteRealmAuditLog has stale data, it means the server
stopped or never uploaded data. We raise MissingDataError in such
cases when a user action led to calculating licenses count from
stale data.
* Reformat "This is a legacy plan" notice on billing page.
* Add a link to the plan name on upgrade page title.
* Tweak discount style on billing page.
* Add line break to server login page title.
* Match server login page title and tab title.
* For free trial, don't show number of licenses for current billing period.
* For free trial scheduled to downgrade, don't show number of
licenses for next billing period.
This commit moves the 'update_license_ledger_if_needed' and its
helper function 'update_license_ledger_for_automanaged_plan'
to the 'BillingSession' abstract class.
This refactoring will help in minimizing duplicate code while
supporting both realm and remote_server customers.
Moves the 'update_license_ledger_for_manual_plan' function
to the 'BillingSession' abstract class.
This refactoring will help in minimizing duplicate code while
supporting both realm and remote_server customers.
Creates a process_support_view_request method for BillingSession
to process the various support requests that relate to the billing
system.
Moves approve_realm_sponsorship, update_realm_sponsorship_status,
and attach_discount_to_realm to this new BillingSession method.
Adds a new abstract property to BillingSession to have a string
value, billing_entity_display_name, to use for support messages
sent when these requests are processed.
The "send_invoice" and "charge_automatically" strings used by stripe
for the `collection_method` are referred to both as the "billing
method" and "billing modality" in the billing code.
Because we send this as data to stripe as either `collection_method`
or `billing_modality`, renames any references that are any form of
"billing method".
Implements a nice redirect flow to give a good UX for users attempting
to access a remote billing page with an expired RemoteRealm session e.g.
/realm/some-uuid/sponsorship - perhaps through their browser
history or just their session expired while they were doing things in
this billing system.
The logic has a few pieces:
1. get_remote_realm_from_session, if the user doesn't have a
identity_dict will raise RemoteBillingAuthenticationError.
2. If the user has an identity_dict, but it's expired, then
get_identity_dict_from_session inside of get_remote_realm_from_session
will raise RemoteBillingIdentityExpiredError.
3. The decorator authenticated_remote_realm_management_endpoint
catches that exception and uses some general logic, described in more
detail in the comments in the code, to figure out the right URL to
redirect them to. Something like:
https://theirserver.example.com/self-hosted-billing/?next_page=...
where the next_page param is determined based on parsing request.path
to see what kind of endpoint they're trying to access.
4. The remote_server_billing_entry endpoint is tweaked to also send
its uri scheme to the bouncer, so that the bouncer can know whether
to do the redirect on http or https.
This default setup will be more realistic, matching the ordinary
conditions for a modern server.
Especially needed as we add bouncer code that will expect to have
RemoteRealm entries for realm_uuid values for which it receives
requests.
This does two important things:
1. Fix return type of get_identity_dict_from_session to correctly be
Optional[Union[RemoteBillingIdentityDict, LegacyServerIdentityDict]].
RemoteBillingIdentityDict is the type in the 8.0+ auth flow,
LegacyServerIdentityDict is the type in old servers flow, where only
the server uuid info is available.
2. The uuid key used in request.session["remote_billing_identities"]
should be explicitly namespaced depending on which flow and type
we're
dealing with - to avoid confusion in case of collisions between a
realm and server that have the same UUID. Such a situation should not
occur naturally and I haven't come up with any actual exploitation
ideas that could utilize this by manipulating your server/realm
uuids, but it's much easier to just not think about such collision
security implications by making them impossible.
We pass `next` parameter with /self-hosted-billing to redirect
users to the intended page after login.
Fixed realm_uuid incorrectly required in remote_realm_upgrade_page.
This commit performs a minor update in 'from_name' text for
'support' and 'sponsorship' emails.
Removes capitalization and adds a comment specifying that the
emails are not user-facing.
Since Customer already stores the realm it is linked to and
customer is always created to store sponsorship request, we directly
use customer to link to the sponsorship data for the realm.
Moves and generalizes `switch_realm_from_standard_to_plus_plan`
in stripe.py to be a more general function for changing a
CustomerPlan to a new and valid tier, `do_change_plan_to_new_tier`.
Adds a helper function with the previous function name to be used
for the support view and management command for changing a realm
from the Standard plan tier to the Plus plan tier.
This commit moves a major portion of the 'update_plan`
view to a new shared 'BillingSession.do_update_plan' method.
This refactoring will help in minimizing duplicate code
while supporting both realm and remote_server customers.
If the update / add card session is successful, return user to
manual license management page if user was on it before clicking
the add / update card button.
This commit moves a major portion of the 'initial_upgrade`
view to a new shared 'BillingSession.do_initial_upgrade' method.
This refactoring will help in minimizing duplicate code
while supporting both realm and remote_server customers.
These new models are incomplete and totally untested, but merging this
will provide valuable scaffolding for doing smaller PRs working on
individual gaps, and reveals a clear set of TODOs/refactoring/model
changes needed to support where want to end up.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
This commit moves the main context creation part of the
'billing_home` view to a new shared
'BillingSession.get_billing_page_context' method.
This refactoring will help in minimizing duplicate code
while supporting both realm and remote_server customers.
Moves the 'make_end_of_cycle_updates_if_needed' function to
the 'BillingSession' abstract class.
This refactoring will help in minimizing duplicate code while
supporting both realm and remote_server customers.
Since the function is called from our main daily billing cron job
as well, we have changed 'RealmBillingSession' to accept 'user=None'
(our convention for automated system jobs).
There is a discrepancy between price per license shown on billing
and upgrade page for annual billing frequency due to difference
in methods used to calculate the amount.
To fix it, we use the same method on both the pages.
I didn't use the helpers.format_money directly here since that would
create a visual delay in showing the price per license which will
not be nice considering that will be the only thing on this page
being updated that way.
The stripe fixture used is same as
free_trial_upgrade_by_card--Customer.retrieve.3.json
Moves the 'process_initial_upgrade' function to the
'BillingSession' abstract class.
This refactoring will help in minimizing duplicate code while
supporting both realm and remote_server customers.
Updates `process_initial_upgrade` to take a plan_tier parameter,
so that information can be specific to the type of BillingSession.
Note that ideally the plan tier would be passed as metadata to the
stripe.checkout.Session, but in order to do so, we need to be able
to update the generated stripe fixtures for tests. So for now, we
set the plan tier directly in the stripe event handler code.
Moves the 'setup_upgrade_checkout_session_and_payment_intent'
function to the 'BillingSession' abstract class.
This refactoring will help in minimizing duplicate code while
supporting both realm and remote_server customers.
Generated with both parellel=1 and generate-stripe-fixtures params
on test-backend locally.
Note that the previous series of commits need these test fixtures
to pass CI. If those are changed, then this commit should be updated
or dropped and replaced with a commit that updates the fixtures
based on the new changes.
Updates generate stripe fixtures to use test credit card strings from
Stripe, instead of creating fake stripe.PaymentMethods for the tests
with the test credit card numbers provided by Stripe.
According to stripe's documentation for attaching payment methods to
customers (see https://stripe.com/docs/api/payment_methods/attach),
payment methods should be attached to customers through a SetupIntent
or PaymentIntent, which is what we do as we process new customers
and accounts.
Updates create_stripe_customer so that it is clear that the payment
method should not be added when we directly create a new stripe
customer.
This calculates the largest amount of messages sent within a month for
the last 3 months. The query is targeted for the specific use-case in
this function - for finding the count for a specific server. For
calculating this in bulk for a large number of remote server an
adapted, bulk query will be needed - rather than running this one in a
loop, which would likely be very inefficient.
Instead of randomly generating an integer for the realm string_id
and realm name use a counter to increment the string ids by one.
This way if a particular realm fails the test, it will be easier
to identify which one failed.
Also, removes unused Customer objects that were being returned on
realm creation in the test, as these were not being checked/used
in the test.
In commit a3093fad9, we made Hamlet a billing admin for testing.
Because the approve sponsorship flow sends messages from the
notification bot to both organization owners and billing admins,
we need to update that test to cover both of those user types.
Moves `update_billing_method_of_current_plan` to the BillingSession
abstract class.
Adds a helper function for support views for the realm case:
`update_realm_billing_method`.
Moves `update_sponsorship_status` to BillingSession abstract class
as `update_customer_sponsorship_status`.
Updates the support views to have a helper for updating this on a
realm: `update_realm_sponsorship_status`.
Makes `approve_sponshorship` an abstract method in BillingSession
abstract base class and moves the implementation for realms to the
RealmBillingSession child class.
Adds `approve_realm_sponsorship` helper function that's used in
the support view and initiates the billing session.
Creates an enum class, AuditLogEventType, and an abstract method in
BillingSession, get_audit_log_event, so that we have an abstraction
for getting the audit log event type since it might be different for
Customer objects with a realm vs a remote_server.
This moves the logic for `attach_realm_discount`, which is used in
the support view, to be in the BillingSession class.
Updates the function name to be `attach_discount_to_customer` so
that the context is generalized vs realm specific.
Updates RealmBillingSession implementation to account for actions
that are initiated by a support admin user.
Also moves the helper function `get_discount_for_realm` that is
only used in support views to `corporate/lib/support.py`.
So that `update_or_create_stripe_customer` can work for Customer
objects with either a realm or remote_server, we create an abstract
base class, BillingSession, and implement a child class for the
current implementation of Customer objects with a realm.
Refactoring `update_or_create_stripe_customer` also moves
`create_stripe_customer` and `replace_payment_method` to the
BillingSession class.
In ensure_customer_does_not_have_active_plan, we were already going
through the Customer table to get/check for an active CustomerPlan.
Now we directly get/check for an active CustomerPlan with via the
Customer, which allows for reusing this function for Customer
objects without a Realm set.
Moves two functions in corporate/lib/stripe.py that are used to
get data for the main installation activity analytics page to a
separate file: corporate/lib/analytics.py.
Also, updates these functions for the possibility of realm being
None for a Customer object.
Fixes two bugs involving organization with
exempt_from_license_number_check enabled:
1. If the organization had e.g. 100 users and upgraded their plan,
specifying 50 licenses, the generated LicenseLedger and thus the
corresponding invoice was still for 100 users.
2. The organization was unable to use the billing/plan endpoint (update
plan endpoint) to make their number of licenses less than the current
number of users.
Organizations with exempt_from_license_number_check are supposed to be
able to declare whatever license number they want, as this attribute
allows having pricing schemes where an organization only pays us for a
subset of their users.
This migration applies under the assumption that extra_data_json has
been populated for all existing and coming audit log entries.
- This removes the manual conversions back and forth for extra_data
throughout the codebase including the orjson.loads(), orjson.dumps(),
and str() calls.
- The custom handler used for converting Decimal is removed since
DjangoJSONEncoder handles that for extra_data.
- We remove None-checks for extra_data because it is now no longer
nullable.
- Meanwhile, we want the bouncer to support processing RealmAuditLog entries for
remote servers before and after the JSONField migration on extra_data.
- Since now extra_data should always be a dict for the newer remote
server, which is now migrated, the test cases are updated to create
RealmAuditLog objects by passing a dict for extra_data before
sending over the analytics data. Note that while JSONField allows for
non-dict values, a proper remote server always passes a dict for
extra_data.
- We still test out the legacy extra_data format because not all
remote servers have migrated to use JSONField extra_data.
This verifies that support for extra_data being a string or None has not
been dropped.
Co-authored-by: Siddharth Asthana <siddharthasthana31@gmail.com>
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
exempt_from_license_number_check was initially added allowing
organizations with it enabled to invite new users above their number of
licenses.
However, an organization with this permission enabled,
cannot upgrade if they weren't on a plan already - because when choosing
Manual license management, you cannot enter a number of licenses lower
than the current seat count. However, an organization like that probably
already has some users that they get free of charge - and thus they need
to be able to enter a lower number of licenses in order to upgrade.
This just replaces the billing/upgrade with the statement that
"Your organization has requested sponsored or discounted hosting.", so
it should include an obvious contact in case the customer wants to amend
something or just bump a request that may have gotten missed.
Previously this was only available on the upgrade page - meaning an
organization that already bought a plan wouldn't be able to request a
sponsorship to get a discount or such, even if qualified.
Updates the message sent by the notification bot when an
organization is approved for full sponsorship on Zulip
Cloud Standard to include a request to list and link to
Zulip on any acknowledgement or sponsorship pages.
Disables submit buttons on billing / upgrade page for demo
organizations since they will need to become permanent
organizations before upgrading to Zulip Cloud Standard.
Also creates an alert banner on the same page that links to
the help center article on demo organizations.
Updates sub-headers on demo organizations help center
article to match link text and to follow general convention
of using imperative verb forms in help center subheaders.
Part of #19523.
Co-authored by: Lauryn Menard <lauryn@zulip.com>
This is an unused argument. We removed it so that we don't
need to create a `TypedDict` and unpack it when calling
the test client methods.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Since `HttpResponse` is an inaccurate representation of the
monkey-patched response object returned by the Django test client, we
replace it with `_MonkeyPatchedWSGIResponse` as `TestHttpResponse`.
This replaces `HttpResponse` in zerver/tests, analytics/tests, coporate/tests,
zerver/lib/test_classes.py, and zerver/lib/test_helpers.py with
`TestHttpResponse`. Several files in zerver/tests are excluded
from this substitution.
This commit is auto-generated by a script, with manual adjustments on certain
files squashed into it.
This is a part of the django-stubs refactorings.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Given that these values are uuids, it's better to use UUIDField which is
meant for exactly that, rather than an arbitrary CharField.
This requires modifying some tests to use valid uuids.
Previously, our Stripe webhook event handler code retrieved the
user's email from Stripe using the stripe.Customer.email attribute.
This led to situations such that whenever the email that Stripe had
did not correspond to a UserProfile in Zulip, the payment flow
failed since we couldn't find a UserProfile associated with the
given email.
Now, we pass in the UserProfile.id in the metadata to Stripe's
checkout Session object, so that we can fetch the correct email
in future Stripe requests.
After the Stripe migration to the hosted checkout page, we were
missing one fixture for the test for switching from Standard to
Plus. This commit updates the fixtures for that particular test.
This is a follow-up to #19752. The tests in that PR did not verify
that the financial math involved worked properly. This commit
improves the existing tests and adds new fixtures to make sure
that the financial math works as expected.
It is confusing to have the plan type constants not be namespaced
by the thing they represent. We already have a namespacing
convention in place for constants, so we should use it for
Realm.plan_type as well.
compute_plan_parameters assumed that the value of
tier is always CustomerPlan.STANDARD. We change that behavior
to make the function handle CustomerPlan.PLUS as well.
We ran into a bug in production caused by two issues:
- Some users came from orgs that didn't have a website and since
the URL field was required, they submitted invalid URLs.
- We didn't properly respond to invalid form submissions, which
led to UnboundLocalError exceptions in another part of the
code.
This commit solves this by doing the following:
- We now allow blank URLs and have a convenient placeholder text
label that tells users that they may leave the URL field blank.
- This commit refactors the code such that invalid form submissions
result in an informative error message about what exactly went
wrong.
These changes are all independent of each other; I just didn’t feel
like making dozens of commits for them.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The locally defined NamedTuple was triggering a mypy caching bug
(https://github.com/python/mypy/issues/10913), and we don’t use the
tuple behavior anyway.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
When calling some functions or assigning values to certain attributes,
the arguments/right operand do not match the exact type that the
functions/attributes expect, and thus we fix that by converting types
beforehand.
This fixes a batch of mypy errors of the following format:
'Item "None" of "Optional[Something]" has no attribute "abc"
Since we have already been recklessly using these attritbutes
in the tests, adding assertions beforehand is justified presuming
that they oughtn't to be None.
This upgrades the Stripe API to the most recent version. Going through
the Git history, it looks like our current API version is at 2019-03-14.
The API version should be manually changed in Stripe dashboard at the same
time as the commit is deployed in production.
Backward incompatible changes that are relevant to our codebase between
(2019-03-14, 2020-08-27].
* 2020-08-27 - The `sources` property on Customers is no longer included by
default.
* 2020-03-02 - Nothing applicable
* 2019-12-03 - The `id` field of all invoice line items have changed and are
now prefixed
with `il_`. We only rely on this while we normalize the fixtures.
* 2019-11-05 - Nothing applicable
* 2019-10-17 - The `billing` attribute on invoices, subscriptions, and
subscription schedules is renamed to`collection_method`. The invoice
change is the one that is relevant to us.
* The customer object’s `account_balance` value has been renamed to
`balance`. Only used for the stubs at the moment.
* 2019-10-08 - Nothing applicable
* 2019-09-09 - Nothing applicable
* 2019-08-14 - Nothing applicable
* 2019-05-16 - Nothing applicable
https://stripe.com/docs/upgrades
Also normalize the following IDs in stripe fixtures
* price_[A-Za-z0-9]{24}
* prod_[A-Za-z0-9]{14}
* pi_[A-Za-z0-9]{24}
* il_[A-Za-z0-9]{24}
Previously, we only downgraded and voided small organizations behind
payments only if they had an active plan.
This left us with a bunch of invoices from small realms which used to
have an active plan. It doesn't make much sense for us to get these
realms to pay the invoices so we have decided to just void them. This
commit voids the open invoices of all the small realms without an active
plan and has the last invoice open. Unlike, the realms with an active
plan we don't email them about us voiding the invoice. It's not super
obvious whether Stripe sends an email to the customer when the Invoice
is voided. But they do get the message that the invoice is voided if
they try to pay the invoice through the hosted invoice page.
This avoid some duplicate code as well as improve the readability since
before we were checking for the expected values far away from the
definition of realm. Now we define the expected values right after the
realm definition which improves the code readability.
Also, this get removes the postfixing of realm variable names with numbers.
The postfixing is kind of mess since if we want add any new realm in between
the realms we need to renumber a lot of realm variables.
An additional check for whether customer.stripe_customer_id is
None is added to the function. That check was not really required before
since all the customers with a plan also have a valid value for
stripe_customer_id. So all the calls to stripe.Invoice.list would have
non None value for customer argument.
Even though that is the case, mypy should still have complained about
the possibility of customer.stripe_customer_id being None when passed to
stripe.Invoice.list as customer paramater since mypy don't know that
customers with a plan will always have a non empty value for
stripe_customer_id. Our stripe stubs expect a non empty value for
the customer parameter of stripe.Invoice.list. This is despite the
fact that stripe.Invoice.list can actually be called with customer set
to None. This returns the invoices from the entire organization.
Though, we still decided to ensure that the value of customer should be
non empty since there is no reason for us to ever call this function
with customer set to None. You can just call the function wuthout the
customer argument instead. So this requirement of a non None customer
paramater is useful for catching bugs.
The reason mypy didn't complain was because the type of
Customer.objects.all() is Any and not QuerySet[Customer]. So mypy has no
idea that customer.stripe_customer_id can be theoratically None even
though it was not possible in this [articular case as explained before.
I verified that this was the reason mypy didn't complain by using the
reveal_type function on Customer.objects.all() and the customer object.
After the refactoring it's super to obvious to mypy that the type of the
customer is Customer since it's mentioned in the function defintion. So it
was able to complain about the possibility of customer.stripe_customer_id
being None after the refactoring.
This is a prep commit for the Stripe checkout migration.
The Stripe migration commit adds a lot of new view functions. Keeping
all of the views in one view file makes it super hard for readbability.
So creating a new views folder and splitting the existing view file into
two so that we minimize the changes in the big migration commit.
We are starting to run into situations where this data could be
quite useful for making future decisions, so it makes to store it
in the database, not just in an email.
This function had a confusing name, which could result in someone
using it unintentionally when they meant do_reactivate_user.
We also add docstrings for both functions.
This should have been in https://github.com/zulip/zulip/pull/18066.
The reason, the tests were not failing inspite of the params being
json encoded was because pretty much all these tests did not test
the functionality of the endpoint. Rather they were testing things
like whether the user has the right to access the endpoint and all.
So the value of the params did not matter.
The only one test which is an exception is test_replace_payment_source.
Even though a json encoded token was passed to an endpoint that
expecteda string, the test continued to work becausethe fixtures were
not updated for the test in that PR, so instead of sending an incorrect
json encoded token to stripe endpoint it was sending the correct string
token. Now that we removed the json.dumps of token, we no longer have to
update the fixtures.
I have run the tests with --generate-stripe-fixtures set to True and all
the tests are passing. Not including the fixture changes since the tests
conntinue to work the same with both the existing and new fixtures.
Aman Agrawal
Lauryn Menard
Mateusz Mandera
Tim Abbott
Karl Stolley
Alya Abbott
Prakhar Pratyush
Anders Kaseorg
Prakhar Pratyush
Alex Vandiver
Ujjawal Modi
Zixuan James Li
Eeshan Garg
Zixuan James Li
Vishnu KS