Backup copy to region failing for 2nd setting

[chris]

Hi,

I have AutomatiCloud v3.2.638.2241 running under a purchased license.

I have the 'default' setting set up for region 'ap-southeast-2' (Sydney). Under this setting I have a number of 'Backup EC2 Instance' jobs defined that also copy the snapshots to region 'ap-northeast-1' (Tokyo). These jobs work fine.

However I also have an instance running in 'eu-west-1' (Ireland) that I also want to backup and copy to Tokyo. I've therefore created a 2nd AutomatiCloud setting 'Ireland' and defined a backup job against it otherwise identically to the others.

This job fails to copy snapshots from Ireland to Tokyo.

I get a series of emails from AutomaticCloud as follows. Note that the instance has 3 volumes.

AutomatiCloud Notification (JOBNAME) - success (x3)
EBS snapshot created for volume ID: vol-326d95b0 Snapshot ID:snap-c0cacbf3 Job Type: Backup EC2 Instance Job Name: JOBNAME Job ID: i-f59167c4-201610250848488110
Host: HOSTNAME
Timestamp: 25/10/2016 11:16:33 AM

AutomatiCloud Notification () - failure (x3)
EBS snapshot not copied. Snapshot ID:snap-c0cacbf3 Job Type: Copy EC2 Snapshot Job Name:
Job ID: AWSCopySnapshotJob-201610251116335208
Host: HOSTNAME
Timestamp: 25/10/2016 11:16:44 AM

AutomatiCloud Notification (JOBNAME) - success
Backup Instance ID:i-f59167c4 completed.
Job Type: Backup EC2 Instance
Job Name: JOBNAME
Job ID: i-f59167c4-201610250848488110
Host: HOSTNAME
Timestamp: 25/10/2016 11:17:47 AM

The service logfile for the same, including service startup, is as follow:

2016-10-25 11:15:01,185 [1] INFO Quartz.Util.DBConnectionManager - Registering datasource 'default' with db provider: 'Quartz.Impl.AdoJobStore.Common.DbProvider'
2016-10-25 11:15:01,191 [1] INFO Quartz.Impl.StdSchedulerFactory - Using default implementation for object serializer
2016-10-25 11:15:01,196 [1] INFO Quartz.Impl.StdSchedulerFactory - Using custom data access locking (synchronization): Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore
2016-10-25 11:15:01,196 [1] INFO Quartz.Impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
2016-10-25 11:15:01,205 [1] INFO Quartz.Core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl
2016-10-25 11:15:01,205 [1] INFO Quartz.Core.QuartzScheduler - Quartz Scheduler v.2.2.3.400 created.
2016-10-25 11:15:01,211 [1] INFO Quartz.Impl.AdoJobStore.JobStoreTX - JobStoreTX initialized.
2016-10-25 11:15:01,213 [1] INFO Quartz.Core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.2.3.400) 'QuartzScheduler' with instanceId 'NON_CLUSTERED'
Scheduler class: 'Quartz.Core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'Quartz.Simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'Quartz.Impl.AdoJobStore.JobStoreTX' - which supports persistence. and is not clustered.

2016-10-25 11:15:01,213 [1] INFO Quartz.Impl.StdSchedulerFactory - Quartz scheduler 'QuartzScheduler' initialized
2016-10-25 11:15:01,213 [1] INFO Quartz.Impl.StdSchedulerFactory - Quartz scheduler version: 2.2.3.400
2016-10-25 11:15:01,304 [16] INFO Quartz.Impl.AdoJobStore.JobStoreTX - Freed 0 triggers from 'acquired' / 'blocked' state.
2016-10-25 11:15:01,310 [16] INFO Quartz.Impl.AdoJobStore.JobStoreTX - Recovering 0 jobs that were in-progress at the time of the last shut-down.
2016-10-25 11:15:01,310 [16] INFO Quartz.Impl.AdoJobStore.JobStoreTX - Recovery complete.
2016-10-25 11:15:01,311 [16] INFO Quartz.Impl.AdoJobStore.JobStoreTX - Removed 0 'complete' triggers.
2016-10-25 11:15:01,311 [16] INFO Quartz.Impl.AdoJobStore.JobStoreTX - Removed 0 stale fired job entries.
2016-10-25 11:15:01,323 [16] INFO Quartz.Core.QuartzScheduler - Scheduler QuartzScheduler_$_NON_CLUSTERED started.
2016-10-25 11:15:01,323 [16] INFO Quartz.Server.QuartzServer - Scheduler started successfully. AutomatiCloudService Version: 3.2.638.2241
2016-10-25 11:15:01,323 [16] INFO Quartz.Server.QuartzServer - Running in time zone: AUS Eastern Summer Time UTC-Offset: 11:00:00
2016-10-25 11:15:01,323 [16] INFO Quartz.Server.QuartzServer - Init License
2016-10-25 11:15:01,324 [16] INFO Quartz.Server.QuartzServer - Licensed for: SURNAME USERNAME@DOMAIN.com.au COMPANYNAME
2016-10-25 11:15:01,325 [16] INFO Quartz.Server.QuartzServer - HWID: HWID
2016-10-25 11:15:30,642 [QuartzScheduler_Worker-1] INFO AWSBackupEC2Job - BackupEC2 job rescheduled
2016-10-25 11:16:33,509 [QuartzScheduler_Worker-2] INFO AWSBackupEC2Job - EBS snapshot created for volume ID: vol-326d95b0 Snapshot ID:snap-c0cacbf3
2016-10-25 11:16:33,535 [QuartzScheduler_Worker-2] INFO AWSBackupEC2Job - CopySnapshot job created for snapshot:snap-c0cacbf3 to region: ap-northeast-1
2016-10-25 11:16:37,128 [QuartzScheduler_Worker-2] INFO AWSBackupEC2Job - EBS snapshot created for volume ID: vol-526d95d0 Snapshot ID:snap-bccb0eb1
2016-10-25 11:16:37,153 [QuartzScheduler_Worker-2] INFO AWSBackupEC2Job - CopySnapshot job created for snapshot:snap-bccb0eb1 to region: ap-northeast-1
2016-10-25 11:16:40,854 [QuartzScheduler_Worker-2] INFO AWSBackupEC2Job - EBS snapshot created for volume ID: vol-d36e9651 Snapshot ID:snap-64bf5556
2016-10-25 11:16:40,877 [QuartzScheduler_Worker-2] INFO AWSBackupEC2Job - CopySnapshot job created for snapshot:snap-64bf5556 to region: ap-northeast-1
2016-10-25 11:16:46,337 [QuartzScheduler_Worker-2] INFO AWSBackupEC2Job - BackupEC2 job rescheduled
2016-10-25 11:17:47,594 [QuartzScheduler_Worker-6] INFO AWSBackupEC2Job - Backup EC2 Instance ID:i-f59167c4 completed.

Is this a bug?

[admin]

Can you please double check that the user you are using in the Europe region has all the required rights.

Stephan

[chris]

Thanks for getting back to me on this Stephan.

I have both my 'default' and 'Ireland' setting using the same Access & Secret Key. This access key maps to a global IAM user with the following permission policy:

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:*",
                "rds:*"
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "IP ADDRESS OF AUTOMATICLOUD HOST"
                }
            },
            "Resource": [
                "*"
            ]
        }
    ]
}

Thus from my understanding I don't think this looks like a permissions problem?

[chris]

Note I tested this again after upgrading to the new version 3.3 and it still has the same problem.

[admin]

This seems to be a bug. I will provide a fix in the next days.

Stephan

[admin]

Can you please try 3.3.779.

Stephan

[chris]

Hi Stephen,

I just tested against the latest version 3.3.780 and it looks like it's working now. Thanks for fixing!

Regards,
Chris